PaperlayerView exact revision on arXiv

Source-linked AI summary

PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav

Ram Ramrakhya, Dhruv Batra, Erik Wijmans, Abhishek Das

arXiv:2301.07302v2cs.LGcs.AIcs.RO

TL;DR

ObjectNav asks an agent to find a target object in a new environment, but behavior cloning generalizes poorly to unseen states and human demonstrations are expensive, while RL needs careful reward design. PIRLNav pretrains with behavior cloning on human demonstrations and then finetunes with RL. It reaches 65.0% success, a 5.0% absolute improvement over the previous state of the art, and human demonstrations outperform the tested automatically generated alternatives after finetuning.

  • Problem

    ObjectNav combines object grounding, semantic search, and memory in unseen environments; behavior cloning can generalize poorly to unseen states and is expensive, while RL requires careful reward engineering.

    Source: §1 · pp. 1–2

  • Method

    PIRLNav pretrains an ObjectNav policy with behavior cloning on human demonstrations, then finetunes it with reinforcement learning using a critic-only phase before training both actor and critic.

    Source: §1 · p. 2

  • Results

    65.0% success rate improves on the previous 60.0% ObjectNav result by 5.0% absolute; human-demo BC→RL also outperforms shortest-path and frontier-exploration BC→RL when BC training success is controlled.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Human demonstrations retain an advantage over shortest-path and frontier-exploration trajectories after RL finetuning, while the returns from adding human BC demonstrations diminish at larger dataset sizes.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

We study ObjectGoal Navigation -- where a virtual robot situated in a new environment is asked to navigate to an object. Prior work has shown that imitation learning (IL) using behavior cloning (BC) on a dataset of human demonstrations achieves promising results. However, this has limitations -- 1) BC policies generalize poorly to new states, since the training mimics actions not their consequences, and 2) collecting demonstrations is expensive. On the other hand, reinforcement learning (RL) is trivially scalable, but requires careful reward engineering to achieve desirable behavior. We present PIRLNav, a two-stage learning scheme for BC pretraining on human demonstrations followed by RL-finetuning. This leads to a policy that achieves a success rate of $65.0\%$ on ObjectNav ($+5.0\%$ absolute over previous state-of-the-art). Using this BC$\rightarrow$RL training recipe, we present a rigorous empirical analysis of design choices. First, we investigate whether human demonstrations can be replaced with `free' (automatically generated) sources of demonstrations, e.g. shortest paths (SP) or task-agnostic frontier exploration (FE) trajectories. We find that BC$\rightarrow$RL on human demonstrations outperforms BC$\rightarrow$RL on SP and FE trajectories, even when controlled for same BC-pretraining success on train, and even on a subset of val episodes where BC-pretraining success favors the SP or FE policies. Next, we study how RL-finetuning performance scales with the size of the BC pretraining dataset. We find that as we increase the size of BC-pretraining dataset and get to high BC accuracies, improvements from RL-finetuning are smaller, and that $90\%$ of the performance of our best BC$\rightarrow$RL policy can be achieved with less than half the number of BC demonstrations. Finally, we analyze failure modes of our ObjectNav policies, and present guidelines for further improving them.

1. Introduction

ObjectNav asks an agent in a new environment to find an instance of a specified object category, combining object recognition, semantic navigation, and memory. PIRLNav combines behavior-cloning pretraining with RL finetuning, improving ObjectNav results while the paper examines demonstration sources, dataset scaling, and policy failures.

1. Introduction

  • ObjectNav requires grounding object categories, using semantic priors to guide search, and tracking visited locations, while BC generalizes poorly to unseen states and is costly to collect.The cited prior dataset contains 80k demonstrations collected over about 2,894 hours of teleoperation at a cost of $50k.Source: §1 · pp. 1–2
  • 65.0% success rate advances ObjectNav state of the art by 5.0% over the previous 60.0% result.PIRLNav uses behavior-cloning pretraining followed by reinforcement-learning finetuning.Source: §1 · pp. 2–3
  • Human demonstrations outperform shortest-path and frontier-exploration demonstrations after BC and RL, including when BC training success is controlled.The paper also reports human-demo advantages on validation subsets favoring shortest-path or frontier-exploration BC.Source: §1 · pp. 2–3
  • RL finetuning yields diminishing returns as the human-demonstration dataset grows, while frontier-exploration BC performance plateaus beyond 70k trajectories.Figure 1 depicts ObjectNav success rates as the number of BC demonstrations increases and shows the RL-versus-BC improvement narrowing at higher demonstration counts.Source: Figure 1 · p. 1, §1 · p. 3
  • Policy failures include missing goal annotations or blocked navigation meshes, between-floor navigation errors, and failures to recognize or distinguish the goal object.Source: §1 · p. 3

2. Related Work

Prior ObjectNav work includes end-to-end RL, modular learning, and imitation learning, while demonstration-based RL finetuning has been studied across several task settings. The paper positions its contribution as an approach for finetuning imitation-pretrained policies for ObjectNav.

2. Related Work

  • ObjectNav approaches include end-to-end RL, modular methods with mapping and exploration/navigation modules, and imitation learning with human demonstrations.The cited examples include object-relation graphs, semantic segmentation, auxiliary prediction tasks, and exploration incentives.Source: §2 · p. 3
  • Demonstration-pretrained policies have been finetuned with on-policy RL [21], while off-policy demonstration methods [35] [38] pose challenges for recurrent policies.DAPG augments the RL objective to stay close to demonstrations, but the paper says its replay and online data requirements are infeasible at their scale.Source: §2 · p. 3
  • For high-dimensional visual observations with actor-critic RL, VPT [21] is the closest cited setup; PIRLNav instead uses PPO [42] rather than PPG [41].The paper states that PPO avoids VPT's compute-expensive KL constraint and performs better on ObjectNav.Source: §2 · p. 3

3. OBJECTNAV and Imitation Learning

ObjectNav evaluates whether an agent can find a specified object using egocentric observations in an unseen environment. This section describes the HM3D demonstration dataset and the behavior-cloning policy trained on it.

3.1. OBJECTNAV

  • An ObjectNav episode succeeds when the agent stops within 1 m of the goal, can view it, and does so within 500 steps; evaluation uses HM3D-Semantics v0.1 and Habitat Challenge splits.The task uses RGB and GPS+Compass observations; although a depth sensor is described, the authors say they do not use it because they found it unhelpful.Source: §3.1 · pp. 3–4

3.2. OBJECTNAV Demonstrations

  • The authors collected 77k HM3D-Semantics v0.1 ObjectNav demonstrations, requiring about 2,378 human annotation hours.Source: §3.2 · p. 4

3.3. Imitation Learning from Demonstrations

  • Behavior cloning trains the policy on demonstration observation-action pairs by minimizing their negative log likelihood, with inflection weighting for timesteps where actions change.The policy uses a CNN+RNN architecture with ResNet50 image features and a two-layer, 2,048-dimensional GRU; RGB inputs receive color jitter and random shifts.Source: §3.3 · p. 4

4. RL Finetuning

RL finetuning is intended to address behavior cloning’s train/test mismatch and the expense of collecting demonstrations. PIRLNav initializes the actor from behavior cloning, learns a critic before interactive training, and evaluates the resulting policy against ablations and prior approaches.

4. RL Finetuning

  • RL finetuning is motivated by behavior cloning’s train/test mismatch and the time and expense of collecting additional human demonstrations.The paper describes RL finetuning as scalable once annotated 3D scans are available.Source: §4 · p. 4

4.1. Setup

  • The RL objective maximizes expected discounted reward using DD-PPO, with a sparse success reward rather than a dense reward.The policy and critic are the actor and value function, respectively; the paper reports that BC pretraining avoids the zero-reward pathology associated with sparse-reward RL from scratch.Source: §4.1 · p. 4

4.2. Finetuning Methodology

  • Because behavior cloning does not train a critic, PIRLNav first learns one from rollouts with the frozen pretrained actor, then unfreezes the actor for interactive finetuning.The critic phase lasts 8M steps; the critic learning rate then decays while the actor learning rate warms up between 8M and 12M steps.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3. Results

  • PIRLNav achieves 2.2% higher Success than VPT on HM3D VAL, with comparable SPL.The comparison uses 20k human demonstrations and implements VPT’s finetuning method within the same DD-PPO framework.Source: §4.3 · p. 5, Table 1 · p. 5
  • Combining critic-only learning, critic learning-rate decay, and actor learning-rate warmup improves success by 9.9% and SPL by 7.3%.The ablations use a policy pretrained on 20k human demonstrations and 300M RL steps.Source: §4.3 · pp. 5–6, Table 2 · p. 5
  • 65.0% success and 33.0% SPL on both TEST-STANDARD and TEST-CHALLENGE, with 70.4% success and 34.1% SPL on VAL, are reported for the two-stage approach.On TEST-STANDARD, the approach also exceeds ProcTHOR by 11 percentage points in success and 1 point in SPL.Source: Table 3 · p. 6, §4.3 · p. 6

5. Role of demonstrations in BC→RL transfer

The section compares demonstration sources for BC pretraining and examines how those choices affect RL-finetuning, including as BC data scales. Human demonstrations outperform shortest-path and frontier-exploration demonstrations, even under matched BC success or evaluation splits selected to favor the alternatives.

5.1. Results with Behavior Cloning

  • At ~12M training steps, BC success was 64.1% on 77k HD, 44.9% on 70k FE, and 6.4% on 240k SP demonstrations.SP uses privileged-map shortest paths; FE captures task-agnostic exploration, while HD captures human-like task-specific exploration.Source: Table 4 · p. 6, §5.1 · p. 6, §5 · p. 6

5.2. Results with RL Finetuning

  • With BC TRAIN success matched at ~48%, RL-finetuned VAL success was 66.1% from HD, 51.3% from FE, and 43.6% from SP.RL-finetuning follows the HD > FE > SP ordering; SP shows high TRAIN but low VAL success.Source: Figure 5 · p. 7, §5.2 · p. 7
  • Even on VAL splits selected where SP or FE BC outperformed HD, RL-finetuning from HD significantly outperformed the corresponding SP or FE policy.HD BC success on these splits was 0% by design.Source: Table 5 · p. 7, §5.2 · p. 7

5.3. Scaling laws of BC and RL

  • RL gains shrink as HD BC data grows: 35k demonstrations reach 66.4% success versus 70.4% from 77k, while FE performance saturates after 70k demonstrations.The authors hypothesize FE saturation reflects a lack of task-specific exploration.Source: §5.3 · pp. 7–8, Figure 6 · p. 8

6. Failure Modes

Failures include dataset issues, inter-floor navigation, and failures to recognize or reach the goal. The authors identify these patterns as targets for possible improvements to the data and agent.

6. Failure Modes

  • About 36% of failures stem from dataset issues: 27% missing object annotations and 9% navmesh problems; another 21% involve failure to navigate between floors.The authors suggest oversampling inter-floor episodes may help with stair navigation.Source: §6 · p. 8
  • Recognition problems account for 25% of failures: 20% when the agent sees but does not approach the goal, and 5% when it confuses similar objects.The authors point to visual-backbone and object-recognition advances as possible remedies.Source: §6 · p. 8
  • Last-mile navigation contributes 11% of failures, including collisions near the goal that prevent the agent from getting close enough.The authors suggest better goal-distance estimators could help.Source: §6 · p. 8
  • Looping accounts for 4% of failures and exploration failure 2%; the authors report that these together make up about 6%.Exploration failure is a catch-all that includes complex navigation environments and early termination.Source: §6 · p. 8

7. Conclusion

PIRLNav combines behavior-cloning pretraining on human demonstrations with RL finetuning for ObjectNav. The authors report state-of-the-art results and analyze demonstration choices, scaling, and failure modes.

7. Conclusion

  • PIRLNav uses BC on 77k human demonstrations followed by RL finetuning, achieving 65% ObjectNav success, 5% above the previous best.The authors report that BC and BC→RL on human demonstrations outperform those trained on shortest paths or frontier exploration, even with matched BC TRAIN success.Source: §7 · p. 8
  • As BC pretraining data grows and BC success rises, improvements from RL finetuning diminish.The conclusion also identifies dataset annotation noise, inter-floor navigation, and goal-object recognition as major failure sources.Source: §7 · p. 8

A.1. DAPG [20]

DAPG pretrains a policy with behavior cloning, then fine-tunes it with an RL objective augmented by demonstration data. Its implementation poses substantial memory and throughput challenges for ObjectNav’s visual demonstrations.

A.1. DAPG [20]

  • DAPG first behavior-clones a policy, then adds a demonstration-based term to the policy-gradient objective during RL fine-tuning.The objective combines gradients from current-policy trajectories with weighted gradients from replayed demonstrations.Source: §A.1 · p. 12
  • DAPG’s decaying demonstration weight is intended to avoid biasing the gradient toward demonstrations late in training.The weighting schedule uses hyperparameters λ0 and λ1 and update counter k.Source: §A.1 · p. 12
  • Storing visual inputs for 77k ObjectNav demonstrations would require 2TB of system memory, while online replay would hurt throughput.The authors estimate that naively switching between policy experience collection and demonstration replay would require twice the current experience-collection time.Source: §A.1 · p. 12

A.2. Feasibility of Off-Policy RL finetuning

Off-policy fine-tuning with recurrent policies presents replay and hidden-state challenges in partially observable settings. The section also reviews imitation-learning alternatives, noting practical constraints on DAgger and inverse reinforcement learning in this setup.

A.2. Feasibility of Off-Policy RL finetuning

  • Off-policy RL methods use replay buffers, but most use feedforward encoders, which are ill-posed for partially observable settings.The general algorithm framework collects trajectories and stores them for replay-buffer training.Source: §A.2 · p. 12
  • Recurrent off-policy training requires stored state-action trajectories, including RNN hidden states, and sequence-based policy updates.The update also raises the question of how to initialize hidden state for a sampled sequence.Source: §A.2 · p. 12
  • Replay strategies trade hidden-state mismatch against computation, sequence-length variability, or correlated updates, leaving embodied off-policy fine-tuning for recurrent policies open for evaluation.Stored state can become stale; burn-in and stored-state combinations mitigate issues but add computation and tuning requirements.Source: §A.2 · pp. 12–13

B. Prior work in Imitation Learning

  • DAgger needs a queryable expert, while applying DAgger or IRL here is constrained by unavailable human queries, visual-data memory demands, and open systems and evaluation problems.The cited prior implementations would require 2TB of system memory for visual demonstration inputs; replaying demonstrations efficiently during reward-model learning is also an open systems problem.Source: B. Prior work in Imitation Learning · p. 13

C. Training Details

The imitation-learning implementation replays demonstrations across parallel workers and trains with supervised policy updates. The experiments use a fixed 500M-step schedule on 64 GPUs with Adam and a decaying learning rate.

C. Training Details

  • Each worker replays demonstration actions for 64 frames across 8 environments in parallel.The implementation uses a distributed behavior-cloning setup.Source: §C.1 · p. 13
  • After collection, the policy receives a supervised update using 2 minibatches.This update follows each worker’s parallel demonstration rollout.Source: §C.1 · p. 13
  • Behavior cloning runs for 500M steps on 64 GPUs with Adam at learning rate 1.0×10^-3, linearly decayed after each policy update.The default hyperparameters are detailed in Table 6.Source: §C.1 · p. 13

C.2. Reinforcement Learning

The RL implementation fine-tunes with PPO and GAE using distributed experience collection. The VPT comparison replaces PPO’s entropy term with a decaying KL penalty against the frozen imitation policy.

C.2. Reinforcement Learning

  • RL fine-tuning uses PPO with GAE, discount factor γ=0.99, and GAE parameter τ=0.95, distributed across 16 workers on 16 GPUs.Workers collect experience in parallel and perform 2 PPO epochs with 2 minibatches per epoch; fine-tuning runs for 300M steps.Source: §C.2 · pp. 13–14

C.3. RL Finetuning using VPT

  • The VPT comparison initializes critic weights to zero and replaces PPO’s entropy term with KL divergence between the frozen BC policy and current policy.The KL coefficient starts at 0.2 and decays by 0.995 after each policy update; the learning rate is 1.5×10^-5 without decay.Source: §C.3 · p. 14

C.4. RL Finetuning Ablations

The ablations compare direct BC-to-RL finetuning with PIRLNav’s two-phase setup and examine early training behavior. Direct BC-to-RL drops before recovering, while the proposed setup avoids a significant training drop.

C.4. RL Finetuning Ablations

  • Direct BC-to-RL training drops early, then recovers as the critic improves; the two-phase setup with a learning-rate schedule avoids a significant drop.Figure 8 reports slightly higher performance after further training for the policy initialized with ObjectNav pretraining.Source: Figure 8 · p. 14, §C.4 · p. 14
  • The ablation table reports RL-finetuning results on HM3D VAL.The setup uses a policy pretrained on 20k human demonstrations and finetuned for 300M steps.Source: Table 8 · p. 14, §C.4 · p. 14
  • Ablation learning rates were tuned for direct BC-to-RL variants, but PIRLNav’s learning-rate parameters were not tuned.The authors hypothesize that tuning PIRLNav’s parameters could improve performance.Source: §C.4 · p. 14
Loading 2301.07302v2…