PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning

Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, Yizhou Wang

arXiv:1808.03405v2cs.CV

TL;DR

Active tracking must jointly locate objects and control a camera, but conventional separated pipelines require difficult tuning, labeling, and real-world trial and error. The paper trains a ConvNet-LSTM end-to-end tracker with deep reinforcement learning in augmented simulators, achieving robust generalization and potential transfer to VOT videos and a real-world robot.

  • Problem

    Conventional active-tracking systems separate object tracking from camera control, making joint tuning difficult while requiring labeling and costly real-world trial and error.

  • Method

    A ConvNet-LSTM trained with deep reinforcement learning maps raw first-person video frames to camera actions in augmented virtual environments.

  • Results

    The tracker generalizes to unseen paths, appearances, backgrounds, and distracting objects, can recover after occasional target loss, outperforms representative passive pipelines, and potentially transfers from simulation to real-world tracking.

  • Takeaways & Limitations

    End-to-end active tracking can be effective without a passive tracker and can support real-world robot deployment after simulator training.

  • Takeaways & Limitations

    The offline VOT evaluation cannot control camera movement, so its practical value remains uncertain; the paper therefore adds real-world robot experiments.

Abstract

from arXiv · show

We study active object tracking, where a tracker takes visual observations (i.e., frame sequences) as input and produces the corresponding camera control signals as output (e.g., move forward, turn left, etc.). Conventional methods tackle tracking and camera control tasks separately, and the resulting system is difficult to tune jointly. These methods also require significant human efforts for image labeling and expensive trial-and-error system tuning in the real world. To address these issues, we propose, in this paper, an end-to-end solution via deep reinforcement learning. A ConvNet-LSTM function approximator is adopted for the direct frame-to-action prediction. We further propose an environment augmentation technique and a customized reward function, which are crucial for successful training. The tracker trained in simulators (ViZDoom and Unreal Engine) demonstrates good generalization behaviors in the case of unseen object moving paths, unseen object appearances, unseen backgrounds, and distracting objects. The system is robust and can restore tracking after occasional lost of the target being tracked. We also find that the tracking ability, obtained solely from simulators, can potentially transfer to real-world scenarios. We demonstrate successful examples of such transfer, via experiments over the VOT dataset and the deployment of a real-world robot using the proposed active tracker trained in simulation.

1 INTRODUCTION

The paper frames active tracking as a unified problem of locating an object while controlling the camera, and proposes an end-to-end deep-reinforcement-learning tracker trained in virtual environments. The tracker generalizes across unseen conditions, can recover after target loss, outperforms several passive-tracker pipelines, and shows potential for transfer to real-world robots.

  • Motivation: Active tracking unifies object tracking and camera control for mobile robots and drones, unlike passive tracking, which assumes the object remains in view.Conventional passive pipelines are difficult to tune jointly and may require bounding-box labeling and expensive real-world camera-control trial and error.
  • Approach: The proposed end-to-end ConvNet-LSTM maps raw video frames directly to camera movement actions through deep reinforcement learning.Training uses virtual environments, A3C, and a customized reward function encouraging the agent to follow the object.
  • Approach: Environment augmentation varies object appearances, backgrounds, and trajectories to improve the tracker’s generalization ability.The training environments are prepared through simulator plug-ins or specialized simulator APIs.
  • Results: The trained tracker robustly handles unseen object paths, appearances, backgrounds, and distracting objects, and can restore tracking after occasional target loss.The reported recovery case includes abrupt object movement.
  • Results: The proposed approach outperforms several representative passive trackers with hand-tuned camera-control modules, indicating that passive tracking is not necessary for active tracking.The paper presents the direct end-to-end approach as effective without claiming superiority over state-of-the-art passive tracking.
  • Transfer: Tracking learned solely in simulators potentially transfers to real-world scenarios, including a TurtleBot that successfully follows targets indoors and outdoors.The paper also evaluates action outputs on offline VOT videos, where camera movement cannot be controlled.

2 RELATED WORK

Prior work treats active tracking as a combination of object tracking and camera control, while reinforcement learning and environment randomization provide relevant foundations. The paper distinguishes its approach by randomizing target motion as well as visual scene properties to support end-to-end transfer from simulation to a real robot.

  • Object Tracking: Active tracking adds camera control to traditional object tracking, whereas conventional solutions use separate tracking and camera-control components.Passive tracking received more attention partly because its problem setting is simpler.
  • Reinforcement Learning: Reinforcement learning learns a policy mapping states to actions while maximizing accumulated environmental rewards.States may be raw camera frames, and actions may be discrete movement instructions.
  • Reinforcement Learning: Deep ConvNets provide image representations that support reinforcement-learning tasks involving raw visual states, including visuomotor control and games.Related work also applies deep reinforcement learning to localization, region proposals, and visual tracking.
  • Environment Augmentation: Environment augmentation and domain randomization have been used in robotics to improve generalization across obstacle avoidance, navigation, manipulation, and visual settings.Prior work reports some real-world deployments without fine-tuning after simulation training.
  • Environment Augmentation: Compared with prior end-to-end tracking work, this paper randomizes target motion parameters such as velocity and trajectory in addition to textures and layouts.The authors report successful deployment of the resulting end-to-end active tracker on a real-world robot without fine-tuning.

3 OUR APPROACH

The approach trains an end-to-end active tracker with reinforcement learning in virtual environments, mapping visual observations to camera actions. It combines temporal state encoding, A3C optimization, customized rewards, and environment augmentation to improve tracking and generalization.

  • Tracking formulation: The tracker observes first-person RGB frames and selects discrete camera actions including turning, moving forward, combined movement, or no-op.The environment returns an updated frame and reward after each action.
  • Simulated environments: The tracker is trained in simulated ViZDoom and Unreal Engine environments because desired end-to-end training is impractical directly in real-world scenarios.ViZDoom supplies customized tracking maps, while Unreal Engine provides more realistic scenes.
  • Reinforcement learning: A3C trains the policy and value function asynchronously across workers that share neural-network parameters.Workers update the shared model every n time steps using an n-step bootstrap procedure.
  • Network architecture: The network encodes each image with convolutional features, then uses a recurrent sequence encoder to represent target motion over time.The recurrent hidden state summarizes prior observations, capturing movement information unavailable from a single frame.
  • Environment augmentation: Environment augmentation varies training conditions to improve the tracker's generalization ability.The paper describes augmentation as a central technique for preparing diverse environments during training.

4 EXPERIMENTAL RESULTS

The experiments evaluate the method across virtual environments and transfer-oriented real-world sequences. They culminate in deployment on a real-world robot to assess practical tracking value.

  • Evaluation settings: The evaluation covers virtual ViZDoom and Unreal Engine environments with separate qualitative evaluation on VOT real-world sequences.The VOT videos are offline, so the tracker is evaluated by comparing its predicted actions with actual camera movements.
  • Transfer evaluation: The experiments investigate whether tracking learned in virtual environments transfers to real-world scenarios.The VOT evaluation is used to examine transfer potential rather than direct camera control.
  • Robot deployment: A real-world robot deployment demonstrates the practical value of the active tracking algorithm.The deployment is presented as the final experimental component.

4.1 Settings

The experiments use simulated ViZDoom and Unreal Engine environments with varied maps, targets, paths, and backgrounds. A ConvNet-LSTM tracker is trained with reinforcement learning and evaluated using accumulated reward and episode length.

  • Environments: ViZDoom training uses one augmented map, while testing uses nine additional maps with pre-specified but sometimes zig-zag target paths.The zig-zag motion is generated by built-in game-engine behavior and adds tracking difficulty.
  • Environments: Unreal Engine training uses Square with target Stefani, while testing varies maps, target identities, and paths across four environments.S1SP1, S1MP1, S1SP2, and S2MP2 isolate or combine scenery, target, and path changes.
  • Metrics: Accumulated Reward and Episode Length quantify tracking quality and episode duration, respectively.Low accumulated reward indicates tracking failure and terminates the episode.
  • Architecture: The tracker uses an image encoder, a 256-unit LSTM, and actor-critic branches producing a six-dimensional discrete policy and a scalar value.The observation is resized to 84×84×3 and transformed into a 256-dimensional feature vector before temporal encoding.
  • Optimization: Training uses a network initialized from scratch, Adam optimization, and a learning rate α = 0.0001.The reported training configuration also uses β = 0.01, γ = 0.99, update frequency n = 20, and a maximum of 100 × 10^6 iterations.

4.2 Active Tracking in The ViZDoom Environment

In ViZDoom, environment randomization substantially improves training and supports robust active tracking across unseen appearances, backgrounds, paths, and distractions. The tracker can also recover after losing the target and outperforms simulated conventional trackers.

  • Training protocols: RandomizedEnv performs significantly better than SingleEnv in the Standard testing environment.RandomizedEnv reaches its best validation result at 48×10^6 iterations, whereas SingleEnv peaks at about 9×10^6.
  • Ablations: Removing the LSTM causes a considerable drop in accumulated reward, while replacing it with a GRU yields comparable performance.A deeper convolutional architecture provides limited improvement while increasing inference cost.
  • Generalization: The tracker generalizes to different target appearances, backgrounds, sharp turns, counterclockwise paths, and nearby distracting objects.It follows abrupt turns, avoids bait objects, and does not appear to memorize a specialized turning direction.
  • Recovery: When the target disappears after an abrupt turn, the tracker searches with successive turn-right actions and resumes stable tracking after rediscovering it.The target is absent from the image from frames #1376 to #1394 and is rediscovered at frame #1395.
  • Comparison: The end-to-end active tracker beats simulated conventional trackers by a significant gap and can recover from missed targets more effectively.The compared passive trackers often lose the target under camera shifts or sudden turns and can hardly recover from failures.
  • Interpretability: Saliency maps show that target pixels dominate the action-relevant importance, indicating that the tracker learns to locate the target.The maps are image-specific and correspond to the actions predicted for each input frame.

4.3 Active Tracking in The UE Environment

In Unreal Engine tests, the tracker generalizes across target appearances, paths, and maps, although jointly changing map, target, and path reduces accumulated reward. It generally matches or outperforms simulated active trackers.

  • Generalization: The tracker generalizes when the target changes from Stefani to Malcom and when the path changes from Path1 to Path2.These comparisons indicate that performance does not depend on a specialized target appearance or memorized path.
  • Generalization: Changing the map, target, and path together lowers accumulated reward but preserves robust tracking with a comparable episode length.The combined variation is tested in S2MP2.
  • Comparison: The proposed tracker outperforms the simulated active tracker in most cases or achieves comparable results when it is not best.The simulated tracker results also illustrate the difficulty of tuning one camera-control module across conditions.

4.4 Transfer Potential in The VOT Dataset

The simulator-trained tracker produces qualitatively appropriate camera actions on passive VOT videos, suggesting transfer potential to real-world observations. However, the videos cannot be controlled by the tracker, so the active-tracking setting is not fully reproduced.

  • Evaluation: The VOT evaluation feeds frames sequentially to the simulator-trained network and compares predicted actions with the visible target position and size.The experiment uses the Woman and Sphere sequences because camera control is unavailable for recorded video.
  • Action behavior: When the target appears on one side of the image, the tracker tends to turn toward that side to pull it toward the center.Green and red action markers represent leftward and rightward turning actions, respectively.
  • Transfer: The qualitative results provide evidence that a tracker learned solely in a virtual environment can map real-world observations to appropriate actions.The authors present this as transfer potential rather than a complete active-tracking demonstration.
  • Action behavior: When the target becomes larger, the tracker more often outputs no-op actions, intending to stop and wait for the target to move farther away.The action groups include Forward, Left, Right, and Stop.
  • Limitation: The VOT experiment has uncertain practical value because the recorded video is passive and cannot be controlled by the tracker.The paper therefore proceeds to a real-world robot deployment experiment.

4.5 Active Tracking in The Real-world Scenarios

The active tracker is transferred from simulation to a real robot using environment randomization and action-space design, then evaluated indoors and outdoors. Enhanced discrete actions are more robust than continuous actions, while both approaches maintain target placement and scale in the image.

  • Simulation-to-real transfer: Environment randomization covers textures, illumination, target trajectories, and target speeds to reduce overfitting and support transfer from simulation.Texture and illumination randomization target appearance, while trajectory and speed randomization expose motion variation during training.
  • Action-space design: The nine-action discrete space adds backward motion so the robot can adapt when the target is too close, whereas continuous control uses linear and angular velocities.The original six discrete actions were extended because they could not adapt to different target speeds.
  • Deployment setup: The tracker is deployed on a TurtleBot in indoor-room and outdoor-rooftop scenarios, with pedestrians serving as tracking targets.The robot uses an RGB-D camera and updates tracker state and actions at 20 Hz.
  • Evaluation metrics: Relative target size and image-center deviation are used to measure how accurately the tracker follows the target.Both metrics are normalized by image dimensions to avoid effects from different image sizes.
  • Quantitative results: The discrete tracker completes all 10 indoor episodes without failures, while the continuous tracker fails 3 of 10 episodes; discrete control also yields smaller deviation.The target-size variance is similar between action spaces, and the outdoor results show a similar robustness pattern.
  • Qualitative results: Both action spaces tend to place the target at the image center and keep its image size constant during real-world transfer.The qualitative sequence shows coordinated turning and forward motion, including continued movement consistent with the target’s direction rather than a naive stop.

5 CONCLUSIONS

The paper presents an end-to-end active tracker trained with deep reinforcement learning instead of separating tracking from camera control. With environment augmentation and suitable action spaces, the approach generalizes to unseen environments and supports successful real-world robotic deployment.

  • Contributions: The proposed system performs active tracking end to end through deep reinforcement learning, avoiding separate passive tracking and camera-control stages.The conclusion describes the approach as an end-to-end active tracker trained in simulation.
  • Contributions: Training in simulators reduces reliance on human labeling and real-world trial-and-error, while the tracker shows good generalization to unseen environments.The conclusion links the simulation-trained tracker with generalization beyond the training environments.
  • Real-world deployment: More advanced environment augmentation and appropriate action spaces enable a robot to perform active tracking in real-world scenarios.The paper reports successful deployment of a robot using the simulation-trained tracker.
Loading 1808.03405v2…