PaperlayerView exact revision on arXiv

Source-linked AI summary

In View or Out of View? A Vision-Aware Filter that Infers and Reacts to the Human's Field of View

Mengxue Fu, Ya-Chuan Hsu, Yinlong Dai, Erdem Bıyık, Dylan P. Losey

arXiv:2610.14577v1cs.RO

TL;DR

Existing planners may ignore the human’s field of view or assume assistance should be visible. This paper infers FOV from human behavior and filters robot actions, with evaluations finding that vision-aware coordination improves effectiveness and preference across cooking and driving while visibility direction matters.

  • Problem

    Human-aware planners often assume the human can see the whole scene or treat the field of view as static, leaving dynamic FOV inference and out-of-view assistance under-addressed.

    Source: §I · p. 1

  • Method

    The robot infers the human’s field of view from observed actions and reweights a base policy’s candidate actions to steer toward or away from that view.

    Source: §I · p. 1, C. Vision-Aware Policy Refinement · p. 4

  • Results

    Across cooking and driving, vision-aware robots were more effective and preferred partners; in Overcooked, the in-view filter raised mean dishes per episode for Bayes from 0.69 to 1.06.

    Source: §I · p. 2, F. Results · p. 5

  • Takeaways & Limitations

    The results indicate that when robots choose well whether to be seen, they become better partners in coordination with humans.

    Source: §VII · p. 8

    The filter assumes access to a dynamics model and an estimated human model, although it does not require task knowledge.

    Source: §VII · p. 8

Abstract

from arXiv · show

During human-robot interaction, sight plays an important role. When the human sees a robot, they can observe the robot's actions and better predict the robot's future behaviors. Conversely, when the human cannot see the robot, they become uncertain about the robot's intent and plans. In this paper, we focus on the human's field of view. Existing human-aware planners either ignore this field of view and assume the human can observe the entire scene, or intentionally design their interactions so the robot is always visible to the human. By contrast, we enable robots to infer what the human has and has not seen during interaction, i.e., the human's field of view. We then apply this estimate as a vision-aware filter on the robot's actions. Given a nominal robot policy and a model of the interaction, we roll out the policy over a short horizon to see how the robot's state updates. We then reweight the actions based on how those actions impact the robot's predicted visibility: higher weights are assigned to actions that bring the robot into (or away from) the human's line of sight. In practice, our framework provides an online estimation and filtering process that repeatedly updates its understanding of the human's field of view and then locally perturbs robot actions based on the human's vision. We evaluate our vision-aware filter on robots that assist humans in collaborative cooking and coordinate with them in driving environments. Across simulations and a 12-participant user study, we find that our vision-aware filter changes both performance and preference: participants completed over three times as many subtasks with in-view assistance while cooking, and had fewer collisions with cars that purposely stayed out of their view while driving.

I. INTRODUCTION

The paper addresses dynamic human field-of-view inference and uses that estimate to adjust robot actions without changing the underlying policy. Its evaluations compare assistance that moves into or out of view across cooking and driving.

I. INTRODUCTION

  • Existing planners often assume humans see the whole scene or use a known static field of view, while visibility-aware assistance typically favors staying in view.The paper identifies dynamic FOV and the possibility that a robot should remain out of sight as practical challenges.Source: §I · p. 1
  • The paper jointly infers the human’s field of view and intended task because ignoring either can lead to misattributing the human’s actions.The motivating driving example describes how vehicles in view may affect the human’s driving decisions.Source: §I · p. 1
  • The vision-aware filter uses estimated FOV and intent to locally adjust a base policy’s actions toward or away from the human’s line of sight.It rolls out candidate actions, predicts visibility, and reweights the base policy’s action distribution; designers tune the steering direction.Source: §I · p. 1, Figure 1 · p. 2
  • The filter is policy-invariant, allowing it to combine with existing human-robot interaction strategies.It modifies the action distribution without changing the underlying policy.Source: §I · pp. 1–2
  • Across cooking and driving simulations and a user study, vision-aware robots were more effective and preferred partners, and moving into view was not always beneficial.The study compares robots that ignore and reason over the human’s FOV.Source: §I · p. 2

II. RELATED WORK

Prior work studies assistive behavior and models of human perception, but often treats the field of view as fixed or visibility as something to increase. The paper highlights the need to evaluate both in-view and out-of-view assistance.

A. Assistive Robot Behavior and the Human’s View

  • Assistive behaviors include legible motion, information-revealing actions, anticipatory actions, subtask allocation, and conservative trajectories, but rarely account for whether the human is currently watching.Prior work commonly assumes the robot is in view when studying how it should move.Source: A. Assistive Robot Behavior and the… · p. 2

B. Models of What the Human Perceives

  • Models of human perception range from heading or co-presence features to geometric FOV cones and occlusion-aware planning, but generally treat FOV as constant.Some approaches use task-specific heuristics to decide whether to enter the human’s view.Source: B. Models of What the Human Perceives · p. 2

C. Human Factors in Visible Assistance

  • The value of visible assistance varies with load, task, expertise, and trust, yet prior planners that account for visibility generally favor increasing it.The paper identifies a lack of controlled studies isolating visibility by holding the task constant and comparing in-view with out-of-view assistance.Source: C. Human Factors in Visible Assistance · p. 2

III. PROBLEM STATEMENT

The problem is to understand how a robot’s placement inside or outside a human’s limited, changing view affects coordination. The proposed filter infers FOV from observed behavior and adjusts a base policy’s actions toward or away from that view.

III. PROBLEM STATEMENT

  • The human perceives only state components inside a limited viewing angle that follows their heading, while the robot observes the full state.The model assumes the FOV angle is fixed during an interaction but unknown to the robot.Source: §III · p. 2
  • The human acts toward a task goal using observation history, and the robot’s actions change the shared state through the environment dynamics.The coordination question is how placing robot actions inside or outside the human’s view affects interaction.Source: §III · pp. 2–3

IV. VISION-AWARE FILTER

  • The robot updates an FOV posterior from human actions, then refines a base policy’s action distribution to steer toward or away from the estimated view.The direction is a tunable cost choice, and the method assumes models of the human policy and environment dynamics.Source: §IV · p. 3

A. Human Model

The robot uses an approximate human policy model to relate observed behavior to the human’s hidden field of view. It updates a belief over candidate FOVs from observed actions and uses that belief to filter robot actions.

A. Human Model

  • The robot uses an approximate human model because the human’s true policy is unknown, representing likely human actions based on interaction history and observations.The implementation approximates the human policy with predefined decision rules; the model may also be learned from recent behavior.Source: A. Human Model · p. 3

B. Inferring Human’s Field of View

  • The robot updates its belief over candidate FOVs according to how likely the observed human action is under each candidate, as predicted by the human model.The resulting belief captures what the robot infers the human can see from how they act, and is used to construct a filter for any base policy.Source: B. Inferring Human’s Field of View · p. 3

C. Vision-Aware Policy Refinement

The filter refines a nominal robot policy using predicted costs conditioned on the human’s inferred field of view. It balances the base policy’s action preferences against visibility and constrains changes to protect task performance.

C. Vision-Aware Policy Refinement

  • The filter aggregates predicted action costs across possible human FOVs using the posterior belief over those hypotheses.The base policy may be a black box and need not reason about the inferred FOV; it assigns zero probability to unsafe actions.Source: C. Vision-Aware Policy Refinement · p. 3
  • The action distribution combines the base policy’s preferences with FOV-conditioned rollout cost, with λ setting the filter’s influence.At λ = 0, the method reduces to the base policy; increasing |λ| gives greater weight to the FOV-conditioned evaluation.Source: C. Vision-Aware Policy Refinement · p. 3
  • Visibility cost estimates the expected fraction of an H-step rollout that the robot spends in the human’s view.The rollout uses the transition model, the candidate action first, the base policy thereafter, and the approximated human model.Source: C. Vision-Aware Policy Refinement · p. 4
  • The filter preserves task performance by restricting candidates to base-policy-supported actions within a bounded margin, and falls back to the base action when constraints fail or FOV belief is diffuse.The filter may otherwise favor lower visibility cost at the expense of the base policy’s task objective.Source: C. Vision-Aware Policy Refinement · p. 4

V. SIMULATION EXPERIMENTS

The simulations test whether steering robots toward or away from a human’s limited field of view affects coordination across collaborative cooking and driving. The domains span discrete and continuous state and action spaces and require different kinds of coordination.

V. SIMULATION EXPERIMENTS

  • The simulations test whether steering toward or away from the human’s field of view affects coordination differently across collaborative domains.The study compares FOV-aware filtering with FOV-agnostic base policies in cooking and driving.Source: §V · p. 4

A. Experimental Domains

  • Overcooked requires a human and robot to deliver as many dishes as possible, with four kitchen layouts that require passing ingredients between separated sections.The human sees within a heading-centered cone, with walls occluding sight.Source: A. Experimental Domains · p. 4
  • Highway places a human-driven ego vehicle in dense traffic controlled by robot policies, which shape how surrounding vehicles share the road.The human aims to travel a fixed distance quickly while avoiding collisions; the robots do not share the human’s task.Source: A. Experimental Domains · p. 4

B. Simulated Humans

The simulated humans act with limited views in cooking and driving, with domain-specific observations and action rules. The experiments pair these simulated partners with five base robot policies spanning rule-based and learned control, and different levels of partner reasoning.

B. Simulated Humans

  • The simulated Overcooked human uses currently visible ingredient locations and a memory buffer retaining observations for 40 timesteps.It selects the highest-priority available subtask from memory and plans with A*, treating never-observed locations as unwalkable.Source: B. Simulated Humans · p. 4
  • The Highway human observes nearby vehicles within its field of view and 40 m sensing distance, and can adjust speed or change lanes.It changes lanes toward a believed-faster free lane unless the maneuver conflicts with a visible vehicle.Source: B. Simulated Humans · p. 4

C. Base Policies

  • The five base policies span rule-based and learned control and none reasons over the human’s field of view.The set comprises Overcooked Greedy, Solo, and Bayes, plus Highway IDM and DQN.Source: C. Base Policies · p. 5

D. Filter Design

The filter forward-simulates candidate base-policy actions under human and environment models, then scores them by expected visibility under the estimated field of view. The evaluation varies domain settings and reports task, safety, efficiency, maneuvering, and computational measures.

D. Filter Design

  • The filter forward-simulates candidate actions under the human and environment models, then chooses the lowest-cost candidate or locally modifies it.Candidate costs reflect expected visibility under the estimated human field of view.Source: D. Filter Design · p. 5
  • Overcooked evaluates the base policy’s three highest-ranked subtasks with 12-timestep rollouts, while Highway adjusts nearby vehicles’ acceleration using 3-second rollouts.In Highway, the filter applies to vehicles within 45 m, leaves lane changes to the base policy, and assumes neighboring vehicles maintain constant velocity during scoring.Source: D. Filter Design · p. 5

E. Evaluation

  • Overcooked tests four layouts, four FOV angles, and three base policies; Highway tests two base policies and three FOV angles, using 20 seeds per combination.Episodes run up to 400 timesteps in Overcooked and 60 simulated seconds in Highway.Source: E. Evaluation · p. 5
  • Evaluation reports dishes delivered in Overcooked and collisions, distance covered, and completed human lane changes in Highway, alongside filter computation and overhead.These measures cover task success, safety, traffic efficiency, induced maneuvering, and real-time computational feasibility.Source: E. Evaluation · p. 5

F. Results

Simulation results show that the preferred visibility direction depends on the domain: in-view assistance improves cooking performance, while out-of-view assistance generally benefits driving. The filter fits both domains’ real-time budgets.

Overcooked

  • In Overcooked, in-view filtering raised mean dishes per episode from 1.16 to 1.40 for greedy, 1.14 to 1.39 for solo, and 0.69 to 1.06 for Bayes.These averages cover layouts and FOVs; Table I reports mean dishes per 400-timestep episode across four layouts and 20 seeds.Source: F. Results · p. 5, Table I · p. 6
  • At a 30° FOV, in-view filtering increased Overcooked deliveries by over 60% relative to each base policy, with smaller relative gains at wider FOVs.Even a 360° FOV leaves walls occluding parts of the kitchen, so it does not provide full observability.Source: F. Results · p. 5

Highway

  • In Highway, out-of-view filtering reduced DQN-traffic collisions from 4.03 to 2.62 per session on average and also helped under IDM at all but its narrowest FOV.Under IDM, it reduced collisions at 60° and 90° while adding roughly 180 m of distance per session; the authors attribute this to traffic spreading out and drivers braking less.Source: F. Results · pp. 5–6, Table II · p. 6
  • In Highway, in-view filtering under IDM increased lane changes from 19.9 to 27.8 per session alongside more collisions, while out-of-view filtering under DQN reduced lane changes from 13.3 to 10.0.Neither direction helped under IDM at 30°; the authors attribute this to the narrow FOV, with collisions mostly involving unseen vehicles.Source: F. Results · p. 6

Computation Time

  • Filtering took 91 ms on average in Overcooked and remained within the real-time budgets of both Overcooked and Highway.Highway action computation was 21 ms with filtering versus 17.5 ms for the IDM base policy; the budgets were around 3 Hz and 15 Hz, respectively.Source: F. Results · p. 6

VI. USER STUDY

In a 12-participant study, in-view assistance performed best in Overcooked, while out-of-view assistance performed best in Highway. Participants’ ratings and rankings also favored different visibility directions across the two domains.

Experimental Setup

  • The study compared base, in-view, and out-of-view policies in both domains, with FOV fixed at 30° or 60° per participant.Twelve participants completed both domains; six were assigned each FOV, and policy order was counterbalanced.Source: §VI · pp. 6–7

Objective Results

  • In Overcooked, participants completed 11.75 subtasks on average with the in-view robot, over three times the 3.17 with the base policy (p < .001).Six of the seven dishes delivered across the study came from the in-view condition.Source: §VI · p. 7, Table III · p. 7
  • In Highway, participants drove 1855 m with out-of-view assistance versus 1779 m with the base policy, with 1.17 versus 1.25 collisions per session.With in-view assistance, participants had 1.42 collisions and drove 1733 m; vehicles moved into their limited sight and obstructed navigation.Source: §VI · p. 7, Table III · p. 7

Subjective Results

  • In Overcooked, the in-view robot rated above the base policy on all five constructs (p < .05 for all) and was ranked best by 10 of 12 participants.Figure 4 presents 7-point Likert ratings with standard-error bars; Figure 5 shows forced-choice rankings.Source: §VI · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
  • In Highway, out-of-view assistance was rated significantly more helpful than the base policy (p = .036), and 7 of 12 participants preferred it.The in-view robot rated lowest on every construct; the out-of-view robot was described as making way for the ego vehicle.Source: §VI · p. 7, Figure 4 · p. 8, Figure 5 · p. 8

VII. CONCLUSION

The filter updates its estimate of the human’s field of view and reweights base-policy actions to steer robot visibility. Its effects differ by task, and the current formulation assumes a constant field of view within an interaction.

VII. CONCLUSION

  • In cooking, maximizing robot visibility resulted in more efficient teamwork; in driving, keeping vehicles out of sight provided a clearer road and enabled faster motions.The authors report that the right visibility strategy depends on the task and domain.Source: §VII · p. 8
  • The filter updates its estimate of the human’s current field of view, then reweights candidate actions by predicted visibility to steer the robot toward or away from sight.It assumes access to dynamics and an estimated human model but requires no task knowledge; a designer parameter sets the visibility direction.Source: §VII · p. 8
  • The current filter assumes the human’s field of view remains constant within an interaction, although real attention shifts with focus and task.Tracking a moving, task-dependent view is identified as a natural next step.Source: §VII · p. 8
  • The authors suggest visibility could be incorporated into end-to-end learning so robot policies decide whether and when to be seen.They conclude that robots make better partners when they make that choice well.Source: §VII · p. 8
Loading 2610.14577v1…