PaperlayerView exact revision on arXiv

Source-linked AI summary

Network Offloading Policies for Cloud Robotics: a Learning-based Approach

Sandeep Chinchali, Apoorva Sharma, James Harrison, Amine Elhafsi, Daniel Kang, Evgenya Pergament, Eyal Cidon, Sachin Katti, Marco Pavone

arXiv:1902.05703v1cs.ROcs.CVcs.LGcs.NI

TL;DR

Robots with constrained onboard resources can benefit from cloud computation, but congested wireless links can add latency or lose data. This paper formulates offloading as a sequential decision problem and applies deep reinforcement learning; in the synthetic-stream experiments, its policy achieves at least 2.6× higher median episode reward than benchmarks. The authors identify hard latency constraints as a possible extension for safety-critical applications.

  • Problem

    Robots have limited onboard compute and storage, while cloud offloading can expose them to latency, information loss, or failure on congested networks.

    Source: §I · p. 1

  • Method

    The paper formulates robot offloading as a Markov Decision Process and applies deep reinforcement learning to choose between onboard processing and cloud offloading.

    Source: §I · p. 2, A. The Robot Offloading Markov… · p. 4

  • Results

    At least 2.6× higher median episode reward than the benchmarks was reported for the RL policy in synthetic input stream experiments.

    Source: A. Synthetic Input Stream Experiments · p. 8

  • Takeaways & Limitations

    The authors conclude that deep reinforcement learning can effectively optimize the general offloading formulation and outperform common heuristics.

    Source: §VI · p. 9

    The formulation treats latency as a cost; safety-critical applications may instead require hard constraints bounding the latency-time distribution.

    Source: §VI · p. 9

Abstract

from arXiv · show

Today's robotic systems are increasingly turning to computationally expensive models such as deep neural networks (DNNs) for tasks like localization, perception, planning, and object detection. However, resource-constrained robots, like low-power drones, often have insufficient on-board compute resources or power reserves to scalably run the most accurate, state-of-the art neural network compute models. Cloud robotics allows mobile robots the benefit of offloading compute to centralized servers if they are uncertain locally or want to run more accurate, compute-intensive models. However, cloud robotics comes with a key, often understated cost: communicating with the cloud over congested wireless networks may result in latency or loss of data. In fact, sending high data-rate video or LIDAR from multiple robots over congested networks can lead to prohibitive delay for real-time applications, which we measure experimentally. In this paper, we formulate a novel Robot Offloading Problem --- how and when should robots offload sensing tasks, especially if they are uncertain, to improve accuracy while minimizing the cost of cloud communication? We formulate offloading as a sequential decision making problem for robots, and propose a solution using deep reinforcement learning. In both simulations and hardware experiments using state-of-the art vision DNNs, our offloading strategy improves vision task performance by between 1.3-2.6x of benchmark offloading strategies, allowing robots the potential to significantly transcend their on-board sensing accuracy but with limited cost of cloud communication.

I. INTRODUCTION

Robots face pressure to limit onboard compute and power, yet cloud offloading can bring latency, information loss, or failure on congested networks. The paper frames offloading as sequential decision-making and uses deep reinforcement learning to balance robot and cloud computation.

I. INTRODUCTION

  • Higher-accuracy cloud models can improve sensing, but local models may be faster and more power-efficient, making the choice a resource–accuracy tradeoff.The framework is motivated by the need to keep onboard resources small while perception and decision-making grow computationally expensive.Source: Figure 1 · p. 1, §I · p. 1
  • Onboard compute and storage constraints motivate offloading, but cloud communication can impose latency, information loss, or failure when networks are congested.High-definition video or LIDAR and multiple robots can quickly overload a network.Source: §I · p. 1
  • Deep reinforcement learning policies address a sequential offloading decision under uncertainty, balancing robot and cloud computation while accounting for network conditions.The paper formulates the problem as a potentially partially observed Markov Decision Process.Source: §I · pp. 1–2
  • The paper identifies under-emphasized communication costs and proposes deep reinforcement learning to trade off robot and cloud computation under diverse network conditions.It presents the formulation as a sequential decision-making problem under uncertainty and offers extensible models for robot/cloud compute and network communication.Source: §I · p. 2

A. Cloud Robotics

Cloud robotics refers to offloading computation to cloud resources, a response to limited onboard capacity. The paper uses vision models to illustrate how model accuracy, size, and speed can trade off.

A. Cloud Robotics

  • Cloud robotics broadly means offloading computation to cloud-based resources, including processing sensory inputs such as video, audio, and LIDAR.The paper cites uses spanning mapping, localization, perception, grasping, control, and speech processing.Source: A. Cloud Robotics · p. 2
  • Vision DNNs illustrate the compute–performance tradeoff: mobile-optimized models can sacrifice accuracy for speed and lower power use compared with more accurate models.The paper names MobileNet and ShuffleNet alongside Mask R-CNN as examples.Source: A. Cloud Robotics · p. 2
  • Table I compares neural networks by accuracy, size, and speed, with accuracy measured as mean average precision on MS-COCO.The supplied passage identifies the table’s measures but does not provide cell values.Source: Table I · p. 2

B. Costs of Offloading

Cloud offloading can reduce onboard requirements, but wireless communication adds latency and network costs that prior approaches often handle with fixed choices or hand-designed heuristics. The paper proposes incorporating input-stream features and network conditions into the offloading decision.

B. Costs of Offloading

  • HD video from one robot can exceed 8 Mbps, while uplink-limited cellular networks often provide only 1–10 Mbps to share across users.Wireless networks can also be stochastic and low-bandwidth.Source: B. Costs of Offloading · p. 2
  • Prior cloud-robotics approaches use predetermined key-frame schedules or hand-designed decision rules, and their heuristics can be difficult to scale beyond specific domains.Related IoT approaches also use key-frame heuristics or split models, which may perform poorly under variable network conditions.Source: B. Costs of Offloading · p. 2
  • The proposed framework incorporates input-stream features and network conditions into the system-level decision of whether to offload.The accompanying LIDAR example illustrates that streaming high-bitrate point clouds can lead to partial delivery or network failure.Source: B. Costs of Offloading · pp. 2–3, Figure 2 · p. 3

C. Case Study: Costs of Offloading LIDAR Data

A ROS-based LIDAR streaming test over WiFi received substantially less data than was sent, and multi-sensor streaming produced severe delay, dropped messages, and outages. These observations motivate sharing network capacity and making cloud queries selectively.

C. Case Study: Costs of Offloading LIDAR Data

  • Network experiments measured effective data rate while streaming LIDAR over WiFi from a Velodyne VLP-16 connected to an NVIDIA Jetson TX2, sending 426 KB average point clouds at 10 Hz via ROS to a central server.Source: C. Case Study: Costs of Offloading… · p. 3
  • 16.4 Mbps median receiver rate versus 33.3 Mbps at the source, with 3.92 Hz received sampling versus 10 Hz produced, was measured for ROS LIDAR streaming over WiFi.The authors attribute the low received rate to ROS handling inefficiencies on the tested link.Source: C. Case Study: Costs of Offloading… · p. 3
  • Streaming data from several Velodyne sensors over a previously uncongested WiFi network produced severe delay, dropped ROS messages, and network outages before rigorous measurements could be taken.The authors note that multiple robots can worsen network congestion.Source: C. Case Study: Costs of Offloading… · p. 3
  • The paper proposes combining local compute, active data filtering, and beneficial cloud queries to help robots share networks and reduce congestion-related response latency.The authors connect this framework to swarms of robots using cloud resources.Source: C. Case Study: Costs of Offloading… · p. 3

III. PROBLEM STATEMENT

The robot-offloading problem models a robot processing a stream of sensory inputs, such as video or LIDAR, with a choice between onboard processing and cloud offloading. The formulation captures model confidence, input properties, and a finite cloud-query budget as factors in sequential decisions.

III. PROBLEM STATEMENT

  • The robot processes a sequence of sensory inputs that may have properties such as temporal coherence, which can guide offloading decisions.The paper notes that modeling coherence can be difficult and motivates model-free decision rules.Source: §III · pp. 3–4
  • Cloud and robot models predict the output for an input and provide confidence scores, while the cloud model is typically more accurate and the robot model more computationally efficient.Prediction quality can be evaluated with a task loss such as cross-entropy or RMSE.Source: §III · p. 4
  • Offloading is constrained by a finite budget of cloud queries over a finite horizon, representing limits such as network bandwidth or rate-limited access to a human operator.Source: §III · p. 4
  • The paper formulates offloading as a sequential decision problem using an MDP, allowing policy objectives to be expressed through a cost function.Source: §III · p. 4

A. The Robot Offloading Markov Decision Process

The Robot Offloading MDP represents the robot's decisions, state, reward, and stochastic dynamics over a finite horizon. Its actions trade off cached predictions, local computation, and cloud queries, with the state tracking input features, prediction ages, and remaining query budget.

A. The Robot Offloading Markov Decision Process

  • The MDP comprises a state space, action space, reward function, stochastic dynamics, and finite problem horizon.The dynamics capture sensory-input evolution and action-dependent changes to stored predictions and query budget.Source: A. The Robot Offloading Markov… · pp. 4–5
  • The four actions reuse the last robot or cloud prediction, or compute a new prediction with the current robot or cloud model.Reusing a prediction exploits temporal coherence; querying the cloud consumes query budget.Source: A. The Robot Offloading Markov… · pp. 4–5
  • The MDP state includes input features, stored predictions, their ages, and the remaining query budget, so the policy can choose among the available actions.The input may be represented by features rather than included directly in the state at full dimensionality.Source: A. The Robot Offloading Markov… · p. 5
  • Input features in the state can include sequential information, a design motivated by measured video-frame coherence and observed LIDAR point-cloud coherence.Source: A. The Robot Offloading Markov… · p. 5
  • The reward trades off prediction loss against computation and network-use costs, with weights and costs chosen to reflect the application and robot constraints.For example, the paper notes that a small-battery robot may assign a higher cost to querying its onboard model.Source: A. The Robot Offloading Markov… · p. 5

B. The Robot Offloading Problem

The Robot Offloading Problem seeks an optimal policy that maximizes expected cumulative reward for given robot and cloud models, query budget, and finite horizon. The formulation accommodates different sensing modalities and cost functions, while the robot can fall back to its local model during network failure.

B. The Robot Offloading Problem

  • The Robot Offloading Problem is to find a policy maximizing expected cumulative reward for specified robot and cloud models, query budget, horizon, and offloading MDP.Source: B. The Robot Offloading Problem · p. 5
  • Separate modular robot and cloud models let designers trade off accuracy, speed, and hardware cost, while local-model fallback provides minimum performance during network failure.Source: B. The Robot Offloading Problem · p. 6
  • Exact MDP solution methods such as dynamic programming are difficult to apply because aspects of the problem, notably sensory-input dynamics, are hard to characterize analytically.Source: B. The Robot Offloading Problem · p. 6
  • The framework is agnostic to sensory modality and supports varied data streams and cost functions, with action spaces extendable for multiple offloading options.Source: B. The Robot Offloading Problem · p. 6

IV. A DEEP RL APPROACH TO ROBOT OFFLOADING

The paper casts offloading as a sequential policy problem and uses deep reinforcement learning to choose when to use robot or cloud computation. A recurrent, model-free policy handles uncertain input dynamics and stochastic costs, and is evaluated against several simple offloading strategies.

  • Model-free reinforcement learning learns offloading decisions from state features without modeling sensory-input dynamics, while a recurrent policy estimates context from recent inputs.The recurrent policy addresses possible non-Markovianity because the state includes features from only the two most recent inputs.Source: §IV · p. 6
  • The policy uses a deep neural network trained with Advantage Actor-Critic (A2C), enabling stochastic-reward handling and inexpensive policy evaluation.Evaluation requires only a neural-network pass, unlike optimization-based action selection such as model predictive control.Source: §IV · p. 6
  • Baselines include random sampling, robot-only processing, and cloud-only periodic queries that hold past predictions between queries.The cloud-only policy samples every query-budget interval and uses its previous cloud prediction in between.Source: §IV · p. 6
  • A confidence-based heuristic offloads the least-confident samples while budget remains, representing a key-frame-selection strategy.The paper contrasts it with the standard all-robot and all-cloud strategies.Source: §IV · p. 6

V. EXPERIMENTAL PERFORMANCE OF OUR DEEP RL OFFLOADER

The evaluation benchmarks deep RL offloading in face detection using simulated image streams and real robot video. Its setup combines an on-robot FaceNet/SVM model, a cloud oracle, and a reward that balances prediction accuracy against computation and query costs.

  • The face-detection scenario uses video frames as inputs and sums absolute differences between sequential frames as the offloading-state feature.The robot model combines pretrained FaceNet embeddings with an SVM, while the cloud model is a high-confidence human oracle.Source: §V · pp. 6–7
  • The reward balances prediction accuracy against robot-computation and cloud-query costs, with cloud queries especially penalized for limited human-operator bandwidth.For the reported cost setup, the authors use accuracy weight 1.0 and cost weight 7.0.Source: §V · p. 7
  • A recurrent LSTM policy processes short state histories, and training used A2C for one million episodes with stable convergence observed after 350,000 episodes.The actor and critic use an LSTM layer followed by a 256-unit fully connected layer.Source: §V · p. 7
  • RL achieved over 2.6× the benchmark policies' performance across diverse test episodes and mixed network conditions.The evaluation covers simulated and hardware settings using state-of-the-art vision DNNs.Source: §V · p. 6, Figure 5 · p. 7

A. Synthetic Input Stream Experiments

In synthetic face streams, the RL offloader outperforms benchmark policies while selectively querying the cloud under varied query budgets. The experiments vary temporal coherence and test on unseen faces, with reward reflecting the balance between prediction accuracy and offloading cost.

A. Synthetic Input Stream Experiments

  • Synthetic episodes varied face-stream coherence by switching among individuals in contiguous intervals, with interval lengths set to 1/10–1/12 of an episode.Each episode had 80 steps, and the query-budget fraction was sampled from 0.10, 0.20, 0.50, 0.70, or 1.0.Source: A. Synthetic Input Stream Experiments · p. 7
  • The test set comprised 100 traces with faces distinct from training, evaluated four times at each query-budget fraction from 0.10 to 1.0.The budgets represent highly constrained through unconstrained network settings.Source: A. Synthetic Input Stream Experiments · p. 7
  • RL attained at least 2.6× higher median episode reward than benchmarks and 0.70× the oracle reward, while sparingly querying the cloud when the robot was uncertain.It achieved better prediction accuracy than all-cloud sampling, which bandwidth limits to sparse queries and held predictions.Source: A. Synthetic Input Stream Experiments · pp. 7–8, Figure 6 · p. 8
  • The oracle is an unattainable upper bound because it knows robot and cloud predictions and future inputs perfectly.Figure 6 breaks mean reward into prediction-accuracy and offloading-cost terms.Source: A. Synthetic Input Stream Experiments · pp. 7–8, Figure 6 · p. 8

B. Hardware Experiments

Hardware experiments run the RL offloader on a Jetson with live video, combining a compact robot model with cloud computation. The policy considers robot confidence and video changes, and achieves higher reward than the all-robot, all-cloud, and confidence-threshold baselines.

B. Hardware Experiments

  • The robot model used a FaceNet embedding and SVM, while the cloud model could use a larger model; the offloader received pixel-difference features rather than full images.Summed frame differences indicate video change and help identify rapidly changing faces.Source: B. Hardware Experiments · p. 8
  • The policy queried the cloud when confidence was low, video was chaotic, or hard-to-detect faces appeared, and cloud-offload decisions were only 51.7% correlated with robot confidence.The policy weighted multiple factors and outperformed a confidence-threshold heuristic.Source: B. Hardware Experiments · p. 8
  • On real streaming video, the offloader earned 1.6× the all-robot reward, 1.35× the all-cloud reward, 1.1–1.5× the confidence-threshold heuristic reward, and 0.82× the oracle reward.The threshold heuristic's performance depends on the faces in the video datasets.Source: B. Hardware Experiments · p. 8
  • The offloader ran on a Jetson TX2 with a 900 KB FaceNet model, roughly an order of magnitude smaller than optimized vision DNNs.The compact model allows robot deployment without interfering with perception or control.Source: B. Hardware Experiments · pp. 8–9

VI. DISCUSSION AND CONCLUSIONS

The paper presents a general MDP formulation of cloud offloading for robotics and demonstrates that deep reinforcement learning can outperform common heuristics. It identifies safety constraints and action-dependent input streams as directions for extending the formulation.

VI. DISCUSSION AND CONCLUSIONS

  • Deep reinforcement learning effectively optimizes the general MDP formulation of robotic cloud offloading and outperforms common heuristics.The authors suggest RL may also be effective for modifications of the stated offloading problem.Source: §VI · p. 9
  • For safety-critical applications such as self-driving cars, latency may need hard constraints, such as bounding its distribution, motivating constrained MDPs.The paper treats latency as a cost in its MDP formulation.Source: §VI · p. 9
  • The formulation assumes input streams are independent of robot decisions, while planning or control actions can affect the streams robots receive.The authors propose considering offloading and network characteristics during action selection as a promising extension.Source: §VI · p. 9

Related papers

Loading 1902.05703v1…