PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Aware Assistive Navigation

Masaki Kuribayashi, Zhongkai Shangguan, Eshed Ohn-Bar

arXiv:2609.05596v1cs.CVcs.RO

TL;DR

Assistive navigation agents must plan not only what to say but when to respond, yet current MLLMs struggle to provide safe, timely guidance. The paper introduces the TIMELI benchmark and trains instruction models with reason and timing supervision. Results show poor off-the-shelf performance, some transfer from synthetic training to real videos, and persistent safety and reasoning challenges.

  • Problem

    Current MLLMs rarely plan whether and when to respond, despite the need for timely guidance to support blind users navigating outdoor environments.

    Source: §I · p. 1

  • Method

    TIMELI combines a video-based assistive-navigation benchmark with instruction generation trained using reason and timing supervision.

    Source: B. The TIMELI Benchmark · p. 3, C. Model Architecture · p. 5

  • Results

    Off-the-shelf models perform poorly and mistime instructions, with Timing F1 near or below 0.50; some models fine-tuned on synthetic data transfer to real-world videos.

    Source: B. Results · p. 6

  • Takeaways & Limitations

    TIMELI provides a video-based benchmark for evaluating time-sensitive, safety-aware navigation instructions in dynamic urban environments.

    Source: §I · p. 2, B. The TIMELI Benchmark · p. 3

    Current MLLMs still struggle with fundamental assistive-navigation concepts, and misleading or poorly timed instructions could create hazardous situations.

    Source: §V · p. 6

Abstract

from arXiv · show

Can interactive vision-and-language agents learn not just what to say but also \textbf{\textit{when}} to say it? Current language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness--poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for Multimodal Large Language Model (MLLM)-based agents, we introduce a large-scale multimodal benchmark for an egocentric, assistive navigation task in complex outdoor environments. Using this benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive agents, we will release our simulation, benchmark, and code (available at the project website: https://timeli-icra.github.io/).

I. INTRODUCTION

Assistive navigation requires MLLMs to decide both what to say and when to intervene, but current models often fail to provide safe, timely guidance. The paper introduces TIMELI, a benchmark and model for evaluating this task, and reports failures by existing models in open- and closed-loop settings, including after synthetic-data fine-tuning.

I. INTRODUCTION

  • Assistive agents must anticipate risks and decide when to intervene or remain silent, because poorly timed or incorrect guidance can distract users or lead them into danger.The introduction gives confusing “left” and “right” instructions and verbose guidance during a crossing as examples.Source: §I · p. 1
  • Current MLLMs are not trained as mobility guides, and common training datasets may include few or no blind individuals.The paper notes that this leaves models without the usability knowledge required for the task.Source: §I · p. 1
  • Even with ample fine-tuning data, most off-the-shelf models perform poorly across key task metrics.The introduction frames TIMELI as a benchmark for evaluating safety-critical temporal reasoning in multimodal systems for blind users.Source: §I · p. 2
  • Models fail to provide effective instructions in both open- and closed-loop settings, even after fine-tuning on synthetic data.The benchmark was developed with mobility guides and blind users to capture context-aware guidance, including when to remain silent at intersections.Source: §I · p. 2

II. RELATED WORK

Prior work largely addresses captioning, static or simplified navigation, and manually crafted or rule-timed assistance rather than continuous, goal-oriented instruction generation. TIMELI targets the combined challenge of deciding what to say and when in dynamic outdoor settings.

II. RELATED WORK

  • Prior video and question-answering work generally does not address planning when to cue a response alongside what to say in dynamic, safety-constrained settings.The paper identifies the need to integrate fine-grained landmarks, 3D spatial planning, safety, and conciseness in real time.Source: §II · p. 2
  • Unlike instruction-following VLN, this task generates guidance for users, while much prior VLN uses static, simplified settings without video reasoning, interaction history, or timing.Commercial step-by-step services also rely on planning and location heuristics rather than accounting for GPS error, preferences, or accessibility needs.Source: §II · p. 2
  • The benchmark compares datasets on temporal information, contextual instructions, dynamic obstacles, and dataset size, and includes real-world data reserved for validation.TIMELI contains 1,233,616 images; its synthetic data varies factors such as weather, time of day, and safety hazards.Source: Table I · p. 2
  • Prior systems may produce an instruction at every state, even when it is unnecessary or dangerous, so the task requires sequential decisions about what and when to say.Those decisions involve the high-level plan, environmental hazards, and path adherence.Source: §II · p. 2
  • Existing assistive systems include manually crafted audio cues and costly remote human guidance, while preliminary MLLM systems use rule-based timing or basic captions.The authors draw on user responses, failure scenarios, and interviews with guides and blind users when developing the simulation and benchmark; users favor timely, prioritized information and silence at intersections.Source: §II · pp. 2–3

III. METHOD

The benchmark evaluates sequential, time-sensitive navigation guidance for a blind person moving outdoors with an assistive system. It assumes a forward-facing RGB camera, noisy location information, and a high-level path planner.

III. METHOD

  • The task assumes an outdoor blind-user navigation scenario using a smartphone or wearable with an RGB camera facing the user’s direction of travel.The system observes the view sequentially and issues guidance instructions.Source: §III · p. 3
  • The system is assumed to have noisy location information and a high-level path planner leading to the destination.The paper formalizes the task, then describes benchmark creation and network optimization.Source: §III · p. 3
  • The benchmark is designed to assess whether MLLMs generate time-sensitive and situationally appropriate navigation instructions for blind users.The passage points to Fig. 2 for an overview of model components.Source: §III · p. 3

A. Task Formulation

At each time step, the agent generates an instruction from egocentric video and language inputs that include task specifications and navigation context. The formulation also uses prior instructions and requires the model to remain silent when guidance is unnecessary or inappropriate.

A. Task Formulation

  • Time-aware instruction generation is formulated as a sequential decision task that produces an output instruction at each time step.The output is a sequence of word tokens drawn from a vocabulary.Source: A. Task Formulation · p. 3
  • At inference, the model receives egocentric video and language input containing user and task specifications, a high-level plan, and prior instructions with timestamps.Video includes the current image and past frames; image-based models use a single frame.Source: A. Task Formulation · p. 3
  • The high-level plan is coarse guidance based on a static, standard-definition map, and the task input also includes a goal waypoint and a command such as “turn left.”The command is computed from the angle between the goal coordinate and the user's heading; the plan has seven possibilities.Source: A. Task Formulation · p. 3
  • The task specification explicitly requires the model to output “None” when user feedback is unnecessary or contextually inappropriate.It also includes the history of prior instructions over a time window, with their timestamps.Source: A. Task Formulation · p. 3

B. The TIMELI Benchmark

TIMELI is a video-based benchmark for time-aware, safety-aware navigation instructions in dynamic urban environments. It combines simulated pedestrian data with a real-world test set and instruction design informed by blind participants and mobility guides.

Design Through Collaboration with Mobility Guides

  • Ten blind participants and local Orientation and Mobility guides informed instruction timing and phrasing, including frequent brief cues, wall-based alignment, and natural distance phrases.Guides avoided numeric distances and confirmed obstacle-free paths.Source: B. The TIMELI Benchmark · p. 3

The TIMELI Benchmark

  • The model overview combines video, a waypoint-derived plan, and task specifications with instruction history, and explicitly supervises instructional reasons.The authors report that reason supervision improves guidance performance.Source: Figure 2 · p. 4

Multimodal Temporal Data Collection

  • TIMELI adapts CARLA for first-person pedestrian simulation across five towns and six weather conditions, recording location, egocentric images, depth, and semantic segmentation at 1 FPS.Pedestrians follow traffic-rule-compliant A* paths from randomly assigned sidewalk locations to destinations.Source: B. The TIMELI Benchmark · pp. 3–4

Time-Aware Navigation Instruction Generation

  • Procedural instructions combine directional cues and environmental descriptions, suppress repeated obstacle alerts except within cane length, and provide warnings around intersections.Directional wording depends on displacement; immediate instructions are issued when a direction change is necessary.Source: B. The TIMELI Benchmark · p. 4

Multimodal Temporal Data Collection

  • A YouTube-video test set was curated with an annotation interface that displayed four consecutive frames or a 1-FPS GIF alongside suggested instructions.Annotators matched scenarios to the user study and annotated navigator direction.Source: B. The TIMELI Benchmark · p. 4

Data Statistics

  • The benchmark contains 67K navigation videos; 79.1% have no instruction, while 20.9% include one.Among spoken instructions, 81.0% direct the pedestrian forward and 17.6% direct a left or right turn.Source: B. The TIMELI Benchmark · p. 4

C. Model Architecture

The model predicts why an instruction is warranted before generating it, and adds auxiliary supervision for instruction timing. Its training objective combines output, reason-prediction, and issuance-classification losses.

Reason Prediction

  • The model restructures its output to predict an instruction’s rationale before generating the instruction, with rationale supervision designed to support decision-making and diagnosis.The reason is computed from simulation state and categorized into eight labels, including remaining silent, junction events, obstacles, direction changes, and stopping.Source: C. Model Architecture · pp. 4–5

Timing Supervision

  • A binary timing-classifier head uses an encoded trace of past instruction issuance and visual features to predict whether to issue an instruction.At inference, a negative timing prediction can trigger an early exit that skips instruction generation, reducing latency and computational overhead.Source: C. Model Architecture · p. 5

Loss Function

  • The generated output includes both reason supervision, which is not announced to the user, and the user-facing instruction.The output is supervised with cross-entropy loss.Source: C. Model Architecture · p. 5
  • The overall loss adds reason-prediction and instruction-issuance classification as auxiliary tasks, weighted with a constant hyperparameter β.Issuance classification uses binary cross-entropy.Source: C. Model Architecture · p. 5

IV. EXPERIMENTS

The experiments evaluate off-the-shelf and fine-tuned models in open- and closed-loop settings.

IV. EXPERIMENTS

  • The experiments cover setup and evaluation metrics, then report results for off-the-shelf and fine-tuned models in open- and closed-loop settings.Source: §IV · p. 5

A. Experimental Setup

Models are tested on held-out CARLA towns, with timing, language, and navigation metrics used to assess performance. The setup also distinguishes fine-tuning variants by the additional context and supervision they receive.

Models

  • Training uses CARLA Town05 and Town10, while testing uses Town02, Town03, and Town04; image-input models receive the current video frame.Models start from open-source checkpoints and are trained using four A6000 GPUs.Source: A. Experimental Setup · p. 5
  • The open-loop table compares off-the-shelf and fine-tuned models; “+” adds instruction history, high-level plan, and reason, while “++” also adds timing supervision.All models have access to goal coordinates, and off-the-shelf outputs are guided through in-context learning.Source: Table II · p. 5

Metrics

  • Timing F1 and AUC use a one-second tolerance; BLEU-4 and ROUGE-L assess instructions only when predicted timing falls within that tolerance.The setup also reports instruction conciseness and closed-loop VLN measures, including success rate, route completion, navigation score, collisions per minute, and instructions per minute.Source: A. Experimental Setup · p. 5
  • Navigation Quality Score is the fifth root of the product of success rate, route completion, navigation score, collision penalty, and instruction efficiency.The collision penalty and instruction efficiency depend on collision and instruction rates; the optimal instruction rate is 20 per minute, with σ = 10 controlling acceptable deviation.Source: A. Experimental Setup · p. 5

B. Results

Off-the-shelf models perform poorly on assistive navigation, while fine-tuning and output or prompt choices improve some metrics; image-based models outperform video counterparts. Some models trained on synthetic data transfer to real-world videos, and a fine-tuned image model achieves up to 50% closed-loop success.

  • Off-the-shelf models have low BLEU-4 scores and Timing F1 near or below 0.50; instruction history, a high-level plan, reason prediction, and timing prediction improve fine-tuned model performance.Image-based models outperform video-based counterparts, which struggle to compress long-term information and attend to correct video elements.Source: B. Results · p. 6
  • Some models fine-tuned only on synthetic data transfer to real-world videos, with less degradation for image models than video models.The evaluation uses open-loop real-world videos; current video MLLMs inefficiently handle longer inputs.Source: B. Results · p. 6, Table III · p. 6
  • Up to 50% success: the LLaVA-v1.6++ image-based model guides users to the goal without repetitive instructions in closed-loop evaluation.Open-loop language-modeling success does not always predict better closed-loop performance for some models.Source: B. Results · p. 6, Table IV · p. 6

V. CONCLUSION

Current MLLMs struggle to determine when and what to instruct, and misleading or poorly timed guidance could be hazardous. The paper focuses on smartphones and identifies evaluation on other robotic platforms as future work.

  • Current MLLMs struggle to determine both when and what to instruct, neglecting temporal and contextual factors needed for assistance.Source: §V · p. 6
  • Misleading or poorly timed AI-generated guidance could lead to hazardous situations, making reliability and safety paramount.Source: §V · p. 6
  • The work focuses on smartphones for scalability and low cost; future evaluation on platforms such as ground robots is needed to validate and broaden applicability.Source: §V · p. 6
Loading 2609.05596v1…