PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Cinematography: Previsualizing and Managing Hallucination Risk in Generative Video Reshooting

Nhan, Tran, Neal Wadhwa, Abe Davis, Stefan Stojanov

arXiv:2610.12455v1cs.HCcs.CV

TL;DR

Generative reshooting can change a camera trajectory after capture, but the requested view may require content the take never recorded, and existing workflows do not show filmmakers that risk while they can still act on it. Hybrid Cinematography evaluates support using an editable shot plan and a proxy of the take before generation. In a study with experienced filmmakers, previsualized risk informed camera decisions and surfaced tensions between creative intent, hallucination, and recorded performance authenticity.

  • Problem

    Generative reshooting can request unseen or insufficiently detailed content, and finished-looking outputs do not reveal which parts reproduce the take versus hallucinate.

    Source: §1 · p. 1, §2.4 · p. 3

  • Method

    Hybrid Cinematography uses an editable shot plan and a proxy of the take to previsualize support for a target move before generation, allowing filmmakers to revise, capture more, or accept hallucination.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    In a two-part study with seven experienced filmmakers, participants used the previs to revise moves and choose targeted capture responses, reported less pressure to execute a perfect take, and revealed tensions around creative intent and authenticity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experienced filmmakers grew more comfortable with generative reshooting when they could evaluate risk on set and retain active control over what the model receives.

    Source: §8 · p. 13

    The previs measures physical support, not final render appearance, and models differ in how effectively they borrow distant frames or synthesize texture.

    Source: §7.4 · p. 12

Abstract

from arXiv · show

On a film set, the camera move is committed during a take. Generative video reshooting lets filmmakers change it afterward, but may require hallucinating unrecorded content, a gap sometimes discovered only after leaving the set. We present Hybrid Cinematography, a workflow that bridges physical capture and generative reshooting to manage hallucination risk while filmmakers can still act on it. Using an editable 3D shot plan and a proxy of the take, our previsualization evaluates hallucination risk in real time. Seeing where the take lacks support, filmmakers can iteratively adjust the plan, explore moves that balance capture and generation, shoot guided pickups, or knowingly accept hallucination. We demonstrate the workflow through a mobile augmented reality application for on-set planning, capture, and review, and an offline pipeline for existing video. A study with experienced filmmakers reveals how previsualizing risk informs camera decisions and exposes tensions between creative intent and generative hallucination.

1 Introduction

Hybrid Cinematography connects physical capture and generative reshooting so filmmakers can assess what a new camera move would require the model to hallucinate before generation. The workflow supports on-set and editing decisions, and a study with experienced filmmakers found that previsualizing risk informed camera choices while surfacing tensions around creative intent and authenticity.

1 Introduction

  • Generative reshooting can render a recorded performance from a revised camera trajectory, but requests for unseen or insufficiently detailed content leave parts of the shot to hallucination.The paper calls this unrecorded demand the shot’s hallucination risk.Source: §1 · p. 1
  • The workflow links an editable 3D shot plan with a proxy of the take to show hallucination risk in real time as filmmakers revise a camera move.This lets them explore trade-offs between physical capture and generation without waiting for the generative model.Source: §1 · p. 2
  • A mobile AR application supports on-set planning, guided capture, and review, while an offline pipeline brings the same previsualization to existing video.The on-set application lets filmmakers revise a move or capture additional footage while the physical scene is available.Source: §1 · p. 2
  • In a two-part study with seven experienced filmmakers, participants revised camera moves and chose targeted capture responses, reporting less pressure to execute a perfect take on set.Their decisions revealed tensions between reducing hallucination risk and preserving creative intent, and between plausible generated content and recorded-take authenticity.Source: §1 · p. 2
  • The workflow’s contributions include distinct visualizations for why a take lacks support, targeted responses to those deficits, and a prototype spanning on-set capture and post-production editing.It also reports findings from a study of how experienced filmmakers use previsualized risk to make camera decisions.Source: §1 · p. 2

2 Related Work

Prior work addresses reshooting, shot planning, capture guidance, and AI-assistance controls, but these strands largely separate capture from generation. Hybrid Cinematography measures a take against a target move before generation and makes the resulting support gaps available to filmmakers while they can still revise or capture.

2 Related Work

  • Reshooting models treat the take as fixed, while capture guidance and previsualization generally do not tell filmmakers what a move will leave the model to hallucinate while the scene is still available.Existing reshooting methods improve use of recorded evidence or generation after capture, but do not evaluate risk early enough to capture missing support.Source: §2 · p. 2, §2.1 · p. 2

2.1 Generative Video Reshooting

  • Geometry-based reshooting masks show where a proxy has holes but do not distinguish unobserved surfaces, insufficient sampling, or missing performance evidence at the requested time.Prior work may improve synthesis or use additional takes, yet does not identify which observations a particular reshoot requires.Source: §2.1 · p. 2

2.2 Shot Authoring and Previsualization

  • Shot-authoring tools support interactive viewpoint choices or recorded-performance staging, but showing a recorded performance from another direction traditionally requires another take.Hybrid Tours lets users combine captured clips; Hybrid Cinematography instead makes the range between captured and target moves continuously navigable within one take, with a risk readout.Source: §2.2 · pp. 2–3

2.3 Capture Guidance

  • Capture-guidance systems typically know the view to acquire before guidance begins, whereas this workflow derives needed views or performance from a target move and its unsupported evidence.Departures from the planned path can provide opportunistic coverage for alternative moves.Source: §2.3 · p. 3

2.4 Steering and Inspecting Generative Assistance

  • Generative reshooting outputs do not distinguish reproduced footage from hallucinated content, while Hybrid Cinematography exposes the consequences of changing the model’s input before generation.Unlike systems that inspect or constrain output after generation, its previs lets filmmakers compare support on an interactive proxy before any frame is generated.Source: §2.4 · p. 3

3 Hybrid Cinematography Workflow

Hybrid Cinematography changes the reshooting process by checking a take’s support for a target move before generation. It uses a persistent editable shot plan and lightweight proxy to make support gaps cheap to inspect and leave decisions about capture or hallucination to the filmmaker.

3 Hybrid Cinematography Workflow

  • The workflow checks a take’s support for a move from a proxy before generation, making the check cheap enough to repeat and early enough to capture more.This changes the reshooting process rather than adding a single interactive tool.Source: §3 · p. 3
  • The workflow draws on familiar production practices such as sensor overlays, proxy editing, pickups, shot lists, and marks on the floor.It leaves filmmakers to decide knowingly what content to leave to the model, since geometry cannot determine what a region means to the shot.Source: §3 · pp. 3–4
  • Its design goals are to previsualize where and why a take fails to support a move, make the range between captured and target moves navigable, and give each deficit a distinct signal and response.A single editable shot plan should guide capture, route pickups, and remain the model’s input, with each edit updating the previs.Source: §3 · pp. 3–4

4 Previsualizing Hallucination Risk

Hallucination risk is assessed by comparing what a target move requests with the visual evidence recorded in a take. The method measures three distinct support deficits on a 3D proxy, evaluates interpolated camera moves, and presents diagnostic signals that filmmakers can use to choose a response.

4.1 What a Take Must Supply for a Move

  • The method decouples camera pacing from performance timing while keeping the source clock unchanged, so changing when the camera reaches a viewpoint does not retime the recorded performance.At output time t, the target camera requests the scene and performance at source time t_s, with t_s = t.Source: §4.1 · p. 4
  • A take can lack compatible viewing directions for surfaces, sufficient sampling density for requested framing, or time-matched evidence of a performer’s action.Static scene evidence accumulates across the take, but a different frame of an actor does not record the same physical action.Source: §4.1 · p. 4

4.2 From Deficits to a Previs

  • A 3D proxy combines an accumulated static point cloud with time-indexed performer point clouds, enabling support checks in milliseconds without running the generative model.The proxy is rendered along a move to produce target-view point projections and binary support masks compatible with downstream reshooting-model conditioning.Source: §4.2 · p. 4, §4 · p. 4
  • The previs interpolates between captured and target camera moves, evaluating coverage, resolution, and performance support at projected points along the path.Coverage tests for compatible source observations, resolution flags a ratio above one as requiring hallucinated detail, and performance measures supported visible performer samples per instant; missing tracking or depth is reported as unknown.Source: §4.2 · p. 4
  • Overlays, timeline lanes, and a 3D trajectory map let filmmakers compare risk across the move continuum, but the signals locate missing support rather than predict how convincingly a model will hallucinate.Keeping deficits distinct matters because later pickups can supply static coverage or resolution but cannot reconstruct an unrecorded moment of live performance.Source: §4.2 · p. 4

5 Workflow Implementation: App and Offline Pipeline

The workflow anchors on-set and offline use in an editable 3D shot plan checked against a proxy of the take.

5.1 Planning the Move in the Scene

  • Filmmakers author camera poses in the scene, then interpolate a smooth path through keyframes or presets.They can retime the move independently of its path and adjust keyframes during review.Source: §5.1 · p. 5

5.2 Capturing With the Plan as Guidance

  • During capture, progressive guidance moves from breadcrumb arrows to framing cues, while review evaluates whatever evidence the camera recorded.RGB video, LiDAR depth, camera poses, and performer tracks populate the proxy for on-device review.Source: §5.2 · p. 5

5.3 Reviewing and Responding on Set

  • Review maps coverage, resolution, and performance gaps to move edits, targeted pickups, or deliberate acceptance; the previs flags missing support, not synthesis quality.Edits to the move or take clear acceptance masks; finalized moves, verified pickups, and confirmed masks leave the set for reshooting.Source: §5.3 · pp. 5–6, Figure 2 · p. 5, Figure 3 · p. 6

5.4 The Previs on Ordinary Video

  • An offline pipeline builds the previs from ordinary video and presents diagnostic overlays in a browser viewer.Across seven sequences, filmmakers use it to weigh revising a move against accepting hallucination when physical pickups are unavailable.Source: §5.4 · p. 6

6 Evaluation

A two-part qualitative evaluation with seven experienced filmmakers examined shot planning, guided capture, and navigation of hallucination risk. Participants used the previs to inform camera decisions, while their choices exposed tensions between creative intent and generated content.

6 Evaluation

  • The two-part qualitative evaluation involved seven experienced filmmakers and examined planning, capture, and risk reasoning without a direct baseline.Part 1 used an offline browser exercise; Part 2 demonstrated planning, guided capture, and review on set.Source: §6 · p. 6, §6.1 · p. 6, §6.2 · pp. 6, 8

6.3 Findings

  • AR keyframe planning was rated a median of 7 of 7, and participants said the shared shot plan helped communicate complex moves across a crew.Participants also said guided capture replaced floor marks or verbal walkthroughs and made physically difficult moves feasible.Source: §6.3.1 · p. 8
  • All seven offline-video examples showed edited orbits turning toward unfilmed performer angles, with risk overlays distinguishing unsupported performer views from missing or coarsely sampled background detail.The figure presents renders of what the take supports, not generated reshoots.Source: Figure 7 · p. 9

6.3.2 Reading Hallucination Risk and Navigating Trade-offs Be-

  • Participants rated the previs 7 of 7 for showing where and why the model must invent content, and used the slider to judge acceptable trade-offs qualitatively.Stopping points reflected judgments about performer identity, facial fidelity, and whether the intended move remained viable—not a numerical threshold.Source: §6.3.2 · p. 10
  • Participants iteratively edited keyframes and swept the slider to inspect updated support, refining paths to reduce hallucination while retaining the intended effect.P7 edited keyframes 25 times to limit generated wall visibility; P6 made 20 adjustments to retain the desired effect with less hallucination.Source: §6.3.2 · p. 10

6.3.3 Selecting Targeted Responses for Distinct Deficits.

  • Tailored response cards let participants preview remedies, and on-set warnings for choosing a pickup, retake, or synthesis received a median rating of 6 of 7.Participants favored targeted pickups for static gaps over repeated full takes; unobserved performer angles instead called for retiming, path edits, or replacement takes.Source: §6.3.3 · pp. 10–11
  • Camera retiming was easy to tune (median 6 of 7), but participants cautioned that automated accelerations through unobserved angles could clash with performers’ dramatic beats.They preferred manually sculpting timing, shortening the spatial move, or filming an intentional retake when dramatic intent was at stake.Source: §6.3.3 · p. 11

6.3.4 Filmmaker Comfort Grew with Active Agency and Control.

  • Comfort with generative reshooting rose for all participants, from a median of 4 before the study to 6 after using the on-set system.Participants attributed the increase to agency and transparency, not uncritical trust in generative models.Source: Figure 8 · p. 10, §6.3.4 · p. 11

7 Discussion

The discussion frames risk previsualization as a way to bring a consequential production decision forward, while helping filmmakers balance model support against creative intent and authenticity. It also identifies practical boundaries in which the workflow and its previews are useful.

7.1 Seeing Hallucination Risk Changes How Filmmakers Shoot

  • Making uncaptured content visible before the set is struck brings the decision to move on into the production loop.Crews can decide whether to revise the move, capture more, or rely on post-production.Source: §7.1 · p. 11
  • Seeing how deviations map to model support shifts on-set pressure from perfect takes toward coverage, with filmmakers choosing where to capture and where to accept synthetic infill.Participants valued retaining agency over what the model received rather than relying on blind trust.Source: §7.1 · p. 11
  • The workflow targets complex or physically constrained moves, where repeated takes are costly; routine setups remain faster with traditional filming and quick retakes.Fast on-device proxy evaluation supplies structural feedback without waiting for live video diffusion.Source: §7.1 · p. 11

7.2 Tension Between Hallucination Risk and Creative Intent

  • Separating spatial path from timing supports schedules that hold for live performance and hurry through unobserved angles, but automated timing can disrupt dramatic pacing.Participants valued supported timing options but preferred manual slider and keyframe adjustments to protect dramatic intent.Source: §7.2 · p. 12
  • A continuous compromise slider lets post-production balance cinematic sweep against footage support, while guided pickups can support an ambitious move on set.The pickups give filmmakers a third option when the target move requires unseen background coverage.Source: §7.2 · p. 12
  • The authors propose continuous controls and separable intent components as a design strategy for creative human-AI tools, allowing creators to author expressive elements and delegate routine infill.They cite spatial path and temporal rhythm as components that can be separated.Source: §7.2 · p. 12

7.3 Tension Between Plausibility and Authenticity

  • Participants accepted infill on textureless walls but protected faces, emotional expressions, and story-critical props, distinguishing plausible synthesis from authentic recorded evidence.They cautioned against generative delegation that displaces the operator’s role or rewrites deliberate on-set choices.Source: §7.3 · p. 12

7.4 Limitations and Future Work

  • The workflow’s previews show physical support rather than final render appearance, and current models vary in their ability to borrow distant frames or synthesize texture.Chunked generation can create boundary discontinuities; regenerating transitional clips bridges seams but adds post-processing time.Source: §7.4 · p. 12

8 Conclusion

Hybrid Cinematography uses a shared proxy and editable shot plan to let filmmakers inspect reshoot risk before generation. The appendices describe how inputs are assembled, paths are interpolated, and chunk seams are repaired.

8 Conclusion

  • Hybrid Cinematography lets filmmakers inspect what a generative reshoot would need to hallucinate and decide whether to revise, retime, capture more, or accept synthesis.The same proxy supports repeated, low-cost review as the plan changes.Source: §8 · p. 13

A Inputs and Outputs of the Workflow

  • On-set capture and ordinary video reconstruction both support review of moves between captured and target trajectories before offline reshooting.On set, the system records camera poses, LiDAR depth, and performer masks; for ordinary video, these are estimated offline.Source: A Inputs and Outputs of the Workflow · p. 15

B Path Representation

  • The path interpolates through approved keyframes with smooth position and orientation, while progress only moves forward so retiming can hold or slow but not reverse.Position uses centripetal Catmull-Rom; orientation uses squad, with shot-type presets adding interior keyframes.Source: B Path Representation · p. 15

C Bridging Chunk Seams

  • Chunked Vista4D generations can show jump cuts at joins; the pipeline replaces frames around each join with one additional transition clip, without blending or training.The transition clip is held to existing video at both ends and free in the middle.Source: C Bridging Chunk Seams · p. 15

D Searching over Timing and Extent

The timing-and-extent search proposes schedules that better align the authored path with recorded support, while preserving deliberate holds and spans. Its proposals expose their consequences for filmmaker approval.

D Searching over Timing and Extent

  • Dynamic programming searches forward-only path-progress and take-time schedules, minimizing camera mismatch, room under-resolution, and performer visibility deficits while freezing holds and hand-authored spans.It may stop short of the path endpoint and offers no proposal if any instant remains unsupported or unknown.Source: D Searching over Timing and Extent · p. 15
  • Figure 11’s authored schedule reaches an unsupported 50◦ view, while the proposal holds, advances through supported views, and stops at 30◦ to lower hallucination risk.The grid plots path progress against output time, with better-supported views at the bottom.Source: D Searching over Timing and Extent · p. 15, Figure 11 · p. 16
  • Figure 12 compares generated reshoots using original and revised timing along the same full camera path.The passage identifies the comparison, but does not state its outcome.Source: D Searching over Timing and Extent · p. 16

E Evaluation Materials and Individual Responses

The appendix describes a two-part study and its response materials, noting that participants encountered the parts and questionnaire sections in a fixed order.

E Evaluation Materials and Individual Responses

E.1 Part 1 Tasks and Scoring

Part 1 used an instructed exercise to assess participants’ interpretation of missing coverage and camera-stop choices, followed by an acceptance task and comfort questions. Closing questions also elicited workflow preferences and individual feature requests.

E.1 Part 1 Tasks and Scoring

  • The initial pin task was an instructed exercise, not an unaided test of detecting generated content.Participants identified unseen content in a source take paired with a generated frame, then saw a mask and annotated face region.Source: §E.1 · p. 16
  • In the second scene, participants pinned missing content, viewed coverage and performer warnings, chose a slider stop, and completed checks comparing interpretation and generation needs.The resolution overlay was disabled.Source: §E.1 · p. 16
  • The acceptance task began at λ=0.5, and each participant’s accepted stop was recorded as their submitted response.Source: §E.1 · p. 16
  • Closing questions asked about preview timing and desired changes, while feature requests were individual suggestions rather than shared themes.Requests included direct path editing, limits on viewpoint departure, paused-frame or generated-option previews, an automatic survey pan, and editing-software integration.Source: §E.1 · p. 16
  • Comfort with generative video reshooting was assessed before and after the exercise on the same seven-point scale.The scale ran from 1 (not at all comfortable) to 7 (completely comfortable).Source: §E.1 · p. 16

E.2 Part 2 Ratings

Part 2 ratings followed participants’ use of the mobile AR application, with questions framed around planning, guidance, review, and workflow integration. A separate retiming example illustrates how previs can distinguish better-supported views from views that require more invention.

E.2 Part 2 Ratings

  • Part 2 ratings covered planning, guidance, review, and future workflow integration, including moves difficult or expensive to execute physically.The table reports medians and observed minimum–maximum; agreement items used a 1–7 scale and comfort used 1–7.Source: §E.2 · pp. 16–17, Table 2 · p. 18
  • Retiming preserved the full path and 3.5-second duration while lingering on better-supported views and whip-panning through poorly supported ones.The plot measures projected point-cloud pixels without evidence, not hallucination error.Source: Figure 12 · p. 17
  • Responses should be read in light of questionnaire examples contrasting story-relevant elements with background geometry and describing expected benefits.Source: §E.2 · p. 17

Related papers

Loading 2610.12455v1…