PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Aerial Cinematography In Unstructured Environments With Learned Artistic Decision-Making

Rogerio Bonatti, Wenshan Wang, Cherie Ho, Aayush Ahuja, Mirko Gschwindt, Efe Camci, Erdal Kayacan, Sanjiban Choudhury, Sebastian Scherer

arXiv:1910.06988v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Safely filming moving actors among obstacles demands coordination of scene context and geometric reasoning, while existing approaches impose restrictions such as prior maps or externally localized targets. The paper combines onboard actor estimation and mapping, long-horizon trajectory optimization, and learned artistic shot selection; field experiments report safe, robust execution of diverse shots and artistically appealing images.

  • Problem

    Manually piloting a UAV while identifying actors, anticipating scene changes, avoiding obstacles, and reaching desired viewpoints can require multiple expert pilots.

    Source: §1 · p. 2

  • Method

    The system combines vision-based actor localization and motion prediction, incremental LiDAR mapping, optimization-based trajectory planning, and deep reinforcement learning for artistic viewpoint selection.

    Source: §11 · p. 41

  • Results

    Field experiments verified safe, robust execution of diverse aerial shots, and the authors report that onboard sensing and computing produced smooth plans and artistically appealing images.

    Source: §9.1 · p. 27

  • Takeaways & Limitations

    The system was evaluated in simulation and real-life scenarios across varied terrain, obstacles, actors, trajectories, and shot types.

    Source: §11 · p. 41

    Real-world map-registration and actor-localization noise reduced transferability of the artistic policy trained in simulation with ground-truth data.

    Source: §10.1 · p. 41

Abstract

from arXiv · show

Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely taxing, often requiring multiple expert human operators. Hence, there is demand for an autonomous cinematographer that can reason about both geometry and scene context in real-time. Existing approaches do not address all aspects of this problem; they either require high-precision motion-capture systems or GPS tags to localize targets, rely on prior maps of the environment, plan for short time horizons, or only follow artistic guidelines specified before flight. In this work, we address the problem in its entirety and propose a complete system for real-time aerial cinematography that for the first time combines: (1) vision-based target estimation; (2) 3D signed-distance mapping for occlusion estimation; (3) efficient trajectory optimization for long time-horizon camera motion; and (4) learning-based artistic shot selection. We extensively evaluate our system both in simulation and in field experiments by filming dynamic targets moving through unstructured environments. Our results indicate that our system can operate reliably in the real world without restrictive assumptions. We also provide in-depth analysis and discussions for each module, with the hope that our design tradeoffs can generalize to other related applications. Videos of the complete system can be found at: https://youtu.be/ookhHnqmlaU.

1 Introduction

Autonomous aerial cinematography must coordinate scene understanding and geometric reasoning while filming moving actors amid obstacles. The paper presents an integrated system combining visual actor estimation, online mapping, long-horizon trajectory optimization, and learned artistic shot selection.

1 Introduction

  • Manually piloting requires operators to identify actors, anticipate scene changes, control the UAV, avoid obstacles, and reach viewpoints simultaneously.The system must also work in unscripted scenarios with limited onboard resources.Source: §1 · p. 2, §1.1 · p. 3

1.1 Challenges

  • The central challenge is coupling contextual decisions about actors, scene evolution, and viewpoints with geometric reasoning about visibility and safe reachability.The challenges include visual localization across viewpoints and lighting, operation in unstructured environments, maintaining visibility while avoiding hazards, and making artistic decisions in real time.Source: §1 · p. 2, §1.1 · p. 3

1.2 Contributions

  • The system combines visual actor localization and trajectory forecasting, incremental signed-distance mapping, covariant-gradient trajectory optimization, and deep reinforcement learning for viewpoint selection.The paper also reports extensive quantitative and qualitative evaluation of the integrated system and its modules in simulation and field tests.Source: §1.2 · p. 3
  • The paper presents a unified architecture and implementation with flight-test evaluations, building on earlier work on individual framework components.Those components include visual actor detection, tracking and heading estimation, online mapping, cinematography planning, and artistic viewpoint selection.Source: §1.2 · pp. 3–4

2 Problem Definition

The problem is to control a UAV filming a moving actor in an unknown environment, balancing safe, smooth motion, visibility, and artistic shot quality. Artistic parameters may be fixed before flight or selected dynamically based on the video feed.

2 Problem Definition

  • The UAV must plan a trajectory while filming an actor whose measured state is used to predict future motion.Actor measurements are obtained onboard from a monocular camera and LiDAR, though external sensors or motion capture could also be used.Source: §2 · p. 4
  • An onboard occupancy grid and signed-distance map represent the environment, treating both occupied and unknown space as potentially inside obstacles.Positive signed distances denote free space; negative values denote occupied or unknown space.Source: §2 · p. 4
  • The trajectory objective balances smoothness, obstacle safety, actor occlusion, and shot quality, with replanning required as the sensed environment and actor forecast change.The component costs penalize jerky motion, unsafe obstacle proximity, actor occlusion, and viewpoints that depart from desired artistic guidelines.Source: §2 · p. 4
  • Artistic parameters can be fixed before takeoff or changed dynamically, making shot selection a sequential decision problem because current camera positions influence future images.The operator’s experience and intuition usually determine the artistic-parameter optimization; the paper describes learning its selection from scene context.Source: §2 · pp. 4–5

3 Related Work

Prior work addresses pieces of aerial cinematography, but the cited systems retain constraints in localization, environmental awareness, planning, or artistic control. This paper situates its approach across visual state estimation, mapping, motion planning, and learned artistic selection.

Virtual cinematography

  • Virtual cinematography methods can optimize or reactively track viewpoints, but assume full environmental knowledge and omit real-world robot physics constraints.Other viewpoint-selection work uses precomputed scenarios, user-specified rules, or formal editing patterns.Source: §3 · p. 5

Autonomous aerial cinematography

  • Earlier aerial-filming systems often rely on RTK GPS or motion capture, omit environmental obstacles from safety reasoning, or operate only at low speed, low clutter, or short planning horizons.The authors also identify a previous method that used a prior elevation map and GPS actor localization.Source: §3 · p. 5
  • Other dynamic-target systems vary in localization and obstacle handling, while the authors’ prior work added visual actor localization and online mapping to an earlier framework.Some approaches account for environmental obstacles and occlusions; others handle only actor-caused collisions and occlusions.Source: §3 · p. 6

Making artistic choices autonomously

  • Autonomous artistic selection requires both an artistic-quality metric and an agent that chooses actions to maximize it; this paper combines learned quality and decision-making ideas for shot-type selection.Prior work studied learned image-quality measures and reinforcement learning for behaviors that are difficult to specify with hand-crafted rewards.Source: §3 · pp. 6–7

Related methods

  • Related work covers online voxel mapping and incremental TSDFs, visual detection and heading estimation, and learning with limited labels or temporal continuity.The cited discussion also notes that standard deep-learning detectors can require powerful GPUs and may generalize poorly to aerial-filming conditions.Source: Online environment mapping · pp. 7–8

4 System Overview

The overview presents a four-module parallel architecture and three design hypotheses guiding the system design. It also describes the hardware and simulation platforms.

4.1 Design Hypotheses

  • Onboard sensors are hypothesized to provide enough actor and obstacle information for localization, mapping, and planning despite noisy measurements.The authors frame onboard sensing as a necessary condition for systems that do not rely on off-board ground truth.Source: §4.1 · p. 8
  • Decoupling gimbal control from motion planning is hypothesized to speed planning and improve robustness to noisy actor measurements.The independent 3-DOF camera controller uses direct image feedback, while the planner benefits from a reduced trajectory state space and longer horizon.Source: §4.1 · p. 8
  • Artistic intent is hypothesized to guide high-level camera positioning without controlling low-level motion, favoring safety, stability, and data-efficient training.The proposed hierarchy uses high-level commands rather than an end-to-end low-level visuomotor policy.Source: §4.1 · p. 9

4.2 System Architecture

  • Vision, Mapping, Planning, and Artistic Shot Selection run in parallel, using camera, LiDAR, and GPS inputs to produce gimbal and flight-controller commands.Vision estimates actor motion and controls the gimbal; mapping supplies environment representations; shot selection chooses a mode; planning computes a trajectory balancing safety, smoothness, shot quality, and occlusion avoidance.Source: §4.2 · pp. 9–10

4.3 Hardware

  • The DJI M210 platform carries a TX2 computer, LiDAR, and gimbaled camera, with a 2.30 kg maximum payload and 13-minute full-payload flight time.The camera gimbal is independently controllable; the listed camera records up to 4K at 60 FPS.Source: Figure 3 · p. 10, §4.3 · p. 10

4.4 Photo-realistic Simulation Platform

  • AirSim provides high-fidelity urban and forest simulations with point clouds, occupancy maps, and actor-pose ground truth for evaluating the vision pipeline.The simulation also supplies training data for shot selection.Source: §4.4 · p. 10, Figure 4 · p. 11

5 Visual Actor Localization and Heading Estimation

The vision module localizes and forecasts the actor while controlling the gimbal to keep the actor framed. It combines detection and tracking, heading estimation, ray-casting, and motion forecasting.

5.1 Detection and Tracking

  • A custom detector and template tracker address onboard resource limits and aerial-image mismatch while providing stable actor measurements.The detector is reinitialized when tracker confidence falls below acceptable limits rather than running detection on every frame.Source: §5.1 · pp. 11–12

5.2 Heading Estimation

  • Aerial heading estimation lacks a suitable existing dataset, and temporal continuity in video provides an unsupervised signal that reduces labeled-data needs and improves generalization.The approach uses the semi-supervised algorithm in both training and testing.Source: §5.2 · p. 13

5.2.1 Defining the loss for temporal continuity

  • The heading model jointly minimizes labeled regression loss and a continuity loss that penalizes larger angle differences between temporally closer samples less than those farther apart.The heading is represented by cosine and sine outputs to avoid ambiguity at the −π/π boundary.Source: §5.2.1 · pp. 13–14

5.2.2 Network structure

  • A compact MobileNet-based CNN processes cropped actor bounding boxes and outputs cosine and sine values for the heading angle.The architecture uses 10 group-wise and point-wise convolutional blocks followed by a convolutional layer and a fully connected layer.Source: §5.2.2 · p. 14

5.2.3 Cross-dataset semi-supervised fine-tuning

  • Training combines aerial-task images with open-dataset images, and the paper evaluates fine-tuning with unsupervised application videos.Source: §5.2.3 · p. 14

5.3 Ray-casting

  • Ray-casting projects the actor bounding box’s center-bottom point onto the mapped terrain to estimate world position, then transforms image-space heading into world coordinates.The heading transformation assumes the camera gimbal’s roll is fixed at zero degrees.Source: §5.3 · p. 15

5.4 Motion Forecasting

  • Actor forecasts use a constant-velocity linear Kalman filter for people and an extended Kalman filter with a kinematic bicycle model for cars and bicycles.Both models use a 10 s prediction horizon.Source: §5.4 · p. 15

6 Online Environment Mapping

The mapping subsystem registers LiDAR data, updates an occupancy grid and signed-distance map, and builds a height map used by actor localization and artistic shot selection.

6.1 LiDAR Registration

  • LiDAR points are registered in world coordinates using the sensor-to-aircraft transform and UAV state estimate; hits and misses then inform occupancy updates.Expected misses caused by reflections from the aircraft are filtered.Source: §6.1 · p. 15

6.2 Occupancy Grid Update

  • For a hit, the endpoint voxel receives an occupied log-odds update and intervening cells receive free-space updates; a miss updates only cells between the sensor and endpoint.Voxel state changes are retained for the subsequent signed-distance update.Source: §6.2 · p. 16

6.3 Incremental Distance Transform Update

  • The incremental distance transform uses obstacle-border cells and assigns positive distances to free voxels and negative distances to unknown or occupied voxels.Borders include LiDAR hits and unknown voxels neighboring free space, allowing the map to distinguish free space from unknown space without treating all unknown cells as confirmed obstacles.Source: §6.3 · pp. 16–17

6.4 Building a Height Map

  • A 2D height map stores moving-average LiDAR-hit heights relative to the UAV’s takeoff position and supports actor ray-casting and online artistic choice selection.Source: §6.4 · p. 17

7 Motion Planning

The planner optimizes a smooth UAV trajectory to balance shot quality, obstacle safety, and actor visibility. It uses continuous trajectory optimization and incrementally updates plans as the actor forecast and environment map change.

7.1 UAV Trajectory Definition

  • The planner outputs a UAV trajectory from the actor forecast and decouples UAV heading from planning by pointing the UAV toward the actor.This removes four degrees of freedom and allows camera control from image feedback without accumulated ray-casting errors.Source: §7.1 · p. 18

7.2 Planner Requirements

  • The planner minimizes a linear combination of smoothness, shot-quality, occlusion, and safety costs, using local trajectory optimization for smooth continuous-space plans that can be updated across cycles.Planning must balance roughly 10 s horizons with frequent replanning and smooth motion.Source: §7.2 · pp. 18–19

7.3.1 Smoothness

  • Smoothness is measured by cumulative trajectory derivatives, with derivatives through third order penalized in practice.The derivative weights are set to 1 and the maximum order to 3.Source: §7.3.1 · p. 19

7.3.2 Shot quality

  • Shot quality penalizes deviations from a desired path parameterized by shot scale, relative yaw, and relative tilt.The parameters are defined for single-actor, single-camera scenarios; the text describes a multi-actor extension.Source: §7.3.2 · pp. 19–20

7.3.3 Safety

  • The safety cost penalizes trajectory points intersecting obstacles and decays with distance up to an obstacle threshold.It is computed using the truncated signed-distance map.Source: §7.3.3 · p. 20

7.3.4 Occlusion avoidance

  • The occlusion cost integrates signed-distance penalties over a 2D manifold connecting UAV and actor positions, producing gradients that can pull the trajectory toward full visibility.The formulation is defined for 3D environments and arbitrary obstacle shapes.Source: §7.3.4 · p. 21, Figure 14 · p. 21

7.4 Trajectory Optimization Algorithm

  • Covariant gradient descent uses the cost gradient and an analytic Hessian approximation, reusing the current trajectory to initialize the next planning problem.A single Hessian inversion outside the optimization loop supports replanning at approximately 5 Hz over a 10 s horizon.Source: §7.4 · p. 22

8 Learning Artistic Shot Selection

The section replaces fixed shot parameters with context-dependent shot selection learned through reinforcement learning. A model-free policy uses scene context and artistic rewards to choose among four shot types.

8.1 Deep Reinforcement Learning Problem Formulation

  • The policy selects left, right, back, or frontal shots from scene context, with each choice setting the relative yaw input to the motion planner.Context combines a local height map, the current shot type, and the number of consecutive steps it has been used.Source: §8.1 · p. 23
  • Because the transition probabilities cannot be modeled explicitly, the system uses a model-free RL method and an action-value function to evaluate shot choices.A deep neural network approximates the action-value function for the large state space.Source: §8.1 · pp. 23–24

8.2 Reward Definition

  • The hand-crafted reward evaluates 180 frames per 6-second control step, scoring shot angle against a 15° optimum and actor screen-space presence, while penalizing repeated shot types quadratically.The angle tolerance is ±10°; actor presence bounds are 0.05–0.10. The ideal shot type lasts 12 seconds (2 steps), and high repetition counts receive quadratic penalties.Source: §8.2 · p. 24

8.3 Implementation Details

  • The DQN uses separate input mini-networks whose outputs are combined to produce Q values, with ReLU activations, Adam, Huber loss, and experience replay.The network architecture is shown in Figure 18.Source: §8.3 · p. 25

9 Experimental Results

Field trials and subsystem evaluations test integrated filming, visual actor localization, and motion planning. The results report robust operation in varied settings, while identifying trade-offs involving mapping, visibility, and planning horizons.

9.1 Integrated System Results

  • Field experiments show the integrated system safely and robustly executed diverse aerial shots across actors and environments, producing smooth plans and artistically appealing images with onboard sensors and computing.The testing facility included varied terrain and obstacles; the reported field results cover fixed and adaptive shots.Source: §9.1 · pp. 26–27

9.2.1 Object detection network

  • The Faster-RCNN detector was selected for filming after open-dataset models showed recall below 33% on aerial footage; the SSD model struggled with small objects.The custom filming data included 120 manually labeled images, and the selected detector used a precision threshold of 0.9.Source: §9.2.1 · pp. 27, 29

9.2.2 Heading direction estimation (HDE) network

  • 72.2% accuracy was achieved by the HDE model fine-tuned on labeled and unlabeled drone-filming sequences, versus 68.1% with labeled samples alone and below 30% for open-dataset baselines.The field-task average angle error was 0.359 rad; semi-supervised fine-tuning improved generalization and stability.Source: §9.2.2 · pp. 30–31

9.2.3 3D pose estimation

  • 3D pose estimation had less than 5.7% error in two field tests tracking actors along a straight path and a circle.The tests compared estimated and ground-truth path lengths and circle diameters.Source: §9.2.3 · p. 32, Figure 27 · p. 32

9.3 Planner Evaluation

  • With online mapping, average planning costs rose slightly and trajectories differed minimally from full-map planning, while planning time doubled but remained low enough for 5 Hz operation.The comparison averaged 140 seconds of flight and approximately 700 planning problems; Figure 28 shows the trajectory comparison.Source: §9.3 · p. 33, Figure 28 · p. 33, Table 9 · p. 33
  • The planner operated in unstructured 3D environments containing obstacles such as wires and poles, beyond prior approaches limited to ellipsoids or height maps.The planner’s reported robustness also covered noisy online maps and actor inputs.Source: §9.3 · pp. 33–35
  • Longer planning horizons produced higher-quality plans, while short horizons reasoned myopically; the 1-second horizon could fail to avoid a mound and become stuck in a local minimum.Table 10 describes the cost and computation-time trade-off as planning horizons increase from 1 to 20 seconds.Source: §9.3 · p. 34, Figure 30 · p. 34, Table 10 · p. 34
  • With 40 random spheres, adding occlusion cost improved actor visibility by over 10% versus pure obstacle avoidance, at the cost of greater average distance from the desired trajectory.The evaluation used randomized clutter environments; Table 11 summarizes planner performance.Source: §9.3 · p. 34, Table 11 · p. 35

9.4 Artistic Shot Selection

The shot-selection policies were trained in simulation and evaluated across environments, in a user study, and in field tests. Learned policies outperformed random choice and produced a middle ground between unstable random shots and static back shots.

9.4.1 Learning an artistic policy

  • Policies achieved their highest rewards in their training environments, and generalization was strongest when training and testing used the same environment category.Neighborhood-trained policies tested on BigMap performed better than BlockWorld-trained policies, which the authors attribute as likely related to BlockWorld’s simpler obstacle geometry.Source: §9.4.1 · p. 37, Table 13 · p. 37
  • Learned policies kept the actor in view by avoiding drastic switches between opposite shot positions, while also switching shots regularly and avoiding high obstacles.The policies were evaluated across BlockWorld, BigMap, and photo-realistic Neighborhood environments.Source: §9.4 · p. 36, §9.4.1 · p. 37

9.4.2 User study results

  • In the user study, policies trained with both reward schemes were rated above random and constant-back-shot policies in every scene, while handcrafted-reward policies ranked higher on average than human-reward policies.Ten participants ranked 30-second clips from five scenes; participant preferences differed in some cases.Source: §9.4.2 · pp. 37–38, Table 14 · p. 39

9.4.3 Extended results in field experiments

  • Field tests found that the trained policy combined periodic viewpoint changes with stable visual tracking, unlike random selection’s repeated actor loss and the visually unappealing static back shot.These shot-selection tests used a pre-mapped environment.Source: §9.4.3 · p. 39

10 Discussion

The discussion draws lessons from system development and describes adaptations for different sensors and UAV platforms. It highlights error propagation, mapping and learning choices, and operational limits.

10.1 Lessons Learned

  • Decoupling motion planning from camera control helps prevent cascading estimation errors from causing the robot to lose the actor.Planning uses actor world-projection estimates, while camera control uses current-image detections as feedback.Source: §10.1 · p. 40
  • Long-range LiDAR can keep the area near the robot mapped, reducing planner waits for map updates; smaller maps could also improve mapping-planning frequency.The system used a 250 × 250 × 100m map; shorter-range sensors may require a smaller local map.Source: §10.1 · p. 40
  • Temporal-continuity regularization let a small labeled dataset and a large unlabeled dataset jointly train the actor-localization network.The discussion identifies consistency across sensory modalities as a possible regularization direction for future work.Source: §10.1 · p. 40
  • Fused IMU and barometer height estimates drift during operations over 5 −10 minutes, degrading pointcloud registration and overall system performance.The authors plan to use LiDAR or visual SLAM for more accurate 3D localization.Source: §10.1 · p. 40
  • Real-world map-registration and actor-localization noise reduced transfer of an artistic policy trained in simulation with ground-truth data.The authors propose adding noise to simulations or training with simulated and real-life data.Source: §10.1 · p. 41

10.2 Adapting Our Work to Different UAVs and Sensors

  • Stereo cameras and depth sensors can replace LiDAR for mapping, but their shorter range must be considered against vehicle speed and obstacle scale to maintain safety.The authors identify LiDAR’s range as hundreds of meters.Source: §10.2 · p. 41
  • The architecture can be adapted to smaller or cheaper platforms through the software interface to the aircraft controller, and to fixed-wing or hybrid aircraft with planner changes.The discussion also identifies visually tracking dynamic targets as a possible use for ground robots.Source: §10.2 · p. 41

11 Conclusion

The conclusion presents a system for autonomous cinematography in unknown, unstructured environments, combining long-horizon trajectory optimization with learned artistic decisions. It reports evaluations in simulation and real-life scenarios and identifies extensions for future work.

11 Conclusion

  • The authors frame cinematography as efficient cost optimization for real-time, long-horizon trajectories under sensor noise, with artistic decisions learned from human feedback.This replaces hand-defined heuristics for guiding UAV motion.Source: §11 · p. 41
  • The system combines vision-based actor localization and motion prediction, incremental LiDAR mapping, long-horizon optimization-based planning, and deep reinforcement learning for viewpoint selection.The planner balances artistic objectives and occlusion avoidance while computing safe trajectories for arbitrary obstacle shapes.Source: §11 · p. 41
  • The system was evaluated in simulation and real-life scenarios across varied terrain, obstacles, actor trajectories and types, and shot types.The conclusion describes the experiments as detailed evaluations of robustness and real-time performance.Source: §11 · p. 41
  • Future work includes multi-drone and multi-actor scenarios, dynamic-scene reconstruction in natural environments, and learning the artistic reasoning behind human choices.The proposed multi-agent extension would add costs for inter-drone collisions, sightlines, and multi-actor coverage.Source: §11 · p. 42

A.1 Smoothness cost

The appendix derives a quadratic smoothness objective from a discretized trajectory representation and notes that its Hessian is independent of the current trajectory.

A.1 Smoothness cost

  • The smoothness cost is written as a quadratic objective using discretized trajectory terms and auxiliary terms derived from K, e, and Δt.The text defines Δt = tf n−1 and describes K as a finite differentiation operator and e as an auxiliary matrix for contour conditions.Source: §A.1 · p. 48
  • The smoothness-cost Hessian does not depend on the current trajectory, a property used to speed up the optimization algorithm.The text also gives analytic expressions for the cost gradient and Hessian.Source: §A.1 · p. 48

A.2 Shot quality cost

The appendix discretizes the shot-quality cost and expresses it as a quadratic objective, with analytic gradient and Hessian expressions independent of the current trajectory.

A.2 Shot quality cost

  • The shot-quality cost is calculated in discrete form and rewritten as a quadratic expression using auxiliary matrices.The displayed passage includes shot-position terms across trajectory samples.Source: §A.2 · p. 49
  • Analytic shot-quality gradient and Hessian expressions are independent of the current trajectory.Source: §A.2 · p. 49
Loading 1910.06988v1…