Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

• The task requires agents to infer goal direction and distance from landmarks and construct spatial understanding without instructions or maps.Landmarks may be invisible in complex urban environments.Source: §1 · p. 1
• React makes decisions from current observations alone, which can lead to repeated actions, circling, and short-sighted choices.Without long-term action sequences, the agent may take more actions than needed.Source: §1 · p. 2
• The proposed workflow combines fine-tuned landmark perception, memory-based reflection, and long-term planning to support consistent long-range navigation.Historical trajectories and observations inform an internal city map; planning decomposes routes into sub-goals using reflections and road connections.Source: §1 · p. 2
• Fine-tuned LLaMA3 achieves nearly a 13% success-rate improvement over base LLaMA3.Source: §1 · p. 2
• The workflow significantly outperforms applicable non-task-specific methods, achieving a 54% average success rate across the four city test sets.Source: §1 · p. 2

2 Related Work

• Urban vision-and-language navigation includes LLM methods for indoor settings and outdoor methods guided by ground-level instructions, unlike this paper’s goal-directed task.Source: §2 · pp. 2–3
• Prior agentic workflows use reflection and structured planning for reasoning tasks, but their use for spatial reasoning in navigation remains underexplored.This work applies perception, reflection, and planning to complex urban navigation.Source: §2 · p. 3

3.1 Task Definition

• The urban environment is modeled as an undirected road graph whose nodes have street-view information and whose edges represent movement paths.Landmarks are isolated vertices, and node positions are fixed relative to one another.Source: §3.1 · p. 3
• The urban navigation task is to find a shortest path from a start node to a goal node specified by goal-to-landmark and landmark-to-landmark relations.At each node, the agent uses street views and road connectivity to choose the next node and infer goal direction and distance.Source: §3.1 · p. 3

3.2 Dataset

• The study also aims to examine LLM spatial cognitive abilities, and builds datasets from CBD road networks in four cities discretized at 50-meter intervals.Each road-network node is associated with street-view images, and well-known buildings serve as landmarks.Source: §3.1 · p. 3, §3.2 · p. 3

4.1 Workflow Overview

• PReP passes visual perception results to reflection, which uses a topological representation of visited nodes, before planning considers the reflected goal and road connections to choose moves.The workflow is described as enabling the agent to form a cognitive map of the urban environment.Source: §4.1 · pp. 3–4

4.2 Perception

• Fine-tuned LLaVA detects landmarks in street views and estimates their direction and distance; the agent combines these estimates with the task description to infer the goal's direction and distance.The authors fine-tune LLaVA-7B because it processes images and text together and its world knowledge may support performance with fewer training samples.Source: §4.2 · p. 4

4.3 Reflection

• Episodic memory stores navigation actions and perceptions, while semantic memory summarizes episodes into an intrinsic map and learned strategies such as detours.The agent can retrieve historical goal inferences and connected-node visit status for reflection and planning.Source: §4.3 · p. 5
• Working memory anticipates goal direction from historical perception and movement, reevaluates the inference when landmarks are visible, and supplies a synthesized inference to planning.This anticipation-reevaluation mechanism is intended to help when landmarks cannot be detected and goal direction is lost.Source: §4.3 · p. 5

4.4 Planning

• Long-term planning uses the reflected goal inference, retrieved memory, and prior plan to break a route into sub-goals; short-term decisions select moves using road connections.The plan is updated after assessing its progress and whether the route should change.Source: §4.4 · pp. 5–6

5.1 Experimental setup

• Each city test set has 100 random start-goal tasks; mean shortest routes are 30 steps (1,500 meters), and exceeding 2.5 times the minimum steps counts as failure.The experiments measure success rate and path-length-weighted success rate (SPL).Source: §5.1 · p. 6

5.2 Main Evaluation

• Comparisons include language-based prompting methods and random/RL baselines; language methods share GPT-4-turbo and hyperparameters, with perception held constant.The RL baseline was trained for 1 million environment steps.Source: §5.2 · p. 6
• PReP yields the best navigation performance across all cities; in Beijing, it reaches 66.68% SR and 48.25% SPL.The comparison includes LLM-free and LLM-based methods.Source: §5.2 · p. 6, Table 1 · p. 7
• On Beijing, full PReP’s success rate exceeds React, w/o Reflection, and w/o Planning by 25%, 19%, and 10%, respectively.Removing reflection produces a larger performance drop than removing planning; the perception component is unchanged across variants.Source: §5.2 · pp. 6–7

5.3 Further Analysis

• On Shanghai, zero-shot LLaVA trails fine-tuned LLaVA by 27% in SR and 23% in SPL; fine-tuned LLaVA averages 4% lower SR and 2.5% lower SPL than oracle perception.The oracle uses ground-truth directions and distances measured by GPS.Source: §5.3 · p. 7, Table 2 · p. 8
• GPT-4-turbo significantly outperforms the other tested LLMs without fine-tuning; fine-tuned LLaMA3 ranks second among the models.The fine-tuning data was generated by GPT-4-turbo.Source: §5.3 · p. 7, Table 3 · p. 8
• Success rate remains relatively stable when goals are as far as 2 kilometers away, while PReP takes about 1.5 times the minimum steps to reach the goal.The authors suggest road-network complexity may matter more than distance for task difficulty.Source: §5.3 · pp. 7–8, Figure 5 · p. 8
• Success rate rises with the proportion of path nodes where landmarks are visible, then gradually converges when visibility exceeds 40%.The authors identify landmark recognition as a key factor in navigation; Beijing has about 50% landmark visibility across nodes, while other test sets can fall below 20%.Source: §5.3 · p. 8, Figure 5 · p. 8

• About 54% success, with performance across four cities and difficulty levels indicating robustness and flexibility.Source: §6 · p. 8
• The workflow combines fine-tuned LLaVA perception, memory-based reflection, and LLM planning without step-by-step instructions or maps.Source: §6 · p. 8
• Training only visual perception makes the approach more data-efficient than RL methods.Source: §6 · p. 8

• Fine-tuned LLaMA-8B performs better than other LLMs but does not reach half the performance of GPT-4-turbo.Source: §7 · p. 9
• The limited test set may cause success-rate fluctuations; a larger, more diverse set could provide a more comprehensive evaluation.Source: §7 · p. 9

A Ethical analysis

• The open-access street-view and map data were gathered in compliance with ethical guidelines, with no personally identifiable information collected.Source: A Ethical analysis · p. 10

B Social impacts

• Navigation technology could support mobility for visually impaired people and disaster relief, while potentially reducing human spatial awareness and problem-solving skills.Source: B Social impacts · p. 10

• The dataset covers central business districts in Beijing, Shanghai, New York, and Paris, each spanning a few kilometers.Source: C Datasets details · pp. 10–11
• Road networks are discretized at 50-meter intervals, and each node is associated with street-view images equal in number to its degree.Source: C Datasets details · p. 11
• Street-view images come from Baidu or Google Map APIs, with a 90° field of view, 20° elevation angle, and 1,024×512-pixel resolution.Source: C Datasets details · p. 11

• The training set comprised 8,000 images of 32 landmarks, manually annotated for visibility and bounding boxes, with API metadata providing landmark-camera distances.The images were split into 80% training and 20% test sets and used to generate question-and-answer dialogue data.Source: §D.1 · p. 11
• About 30,000 dialogue turns were used to fine-tune llava-v1.5-7b on one NVIDIA GeForce RTX A100 GPU in about three hours.The scripts used default parameters.Source: §D.1 · p. 11
• The fine-tuned LLaVA model recognizes and segments landmarks accurately, and its rough distance estimates remain effective in subsequent navigation.Figure 6 presents the distance-estimation results.Source: §D.1 · p. 11, Figure 6 · p. 11

D.2 Fine-tuning LLaMA3-8B

• After fine-tuning, LLaMA3-8B outperforms other LLMs except GPT-4-turbo, while performing similarly to GPT-3.5-turbo on the task.Successful GPT-4-turbo navigation samples were used for fine-tuning, with data from Beijing tested in Shanghai and vice versa, and similarly for New York and Paris.Source: §D.2 · p. 12

E Computational cost

• Each agent step takes about 12 seconds; fine-tuning LLaVA requires one 80G NVIDIA GeForce RTX A100 GPU for about three hours.A fine-tuned LLaVA request-response takes 6–8 seconds on that GPU, while LLM API calls take 2–5 seconds depending on the model.Source: E Computational cost · p. 12

F Case Study

• In the case study, PReP reaches the goal after an initial detour, using landmark observations and history-based reflection to infer direction and re-plan.When landmarks are not visible, the agent can use its trajectory and prior goal inferences to anticipate direction from its current position.Source: F Case Study · p. 12

G.1 Prompt of Full PReP

• Full PReP prompts perception of landmark visibility and distance, reflection on trajectory memory and conflicting goal estimates, and a plan-guided next action.The example estimates the goal by averaging two consistent coordinate inferences while treating other estimates as outliers, then identifies a southward move as the current plan step.Source: §G.1 · pp. 12–13

G.2 Prompt of plain PReP

• Plain PReP receives the current connections and a southward goal inference, then chooses the unvisited south route.The example contrasts that option with a visited north route.Source: §G.2 · p. 13

G.3 Prompt of PReP without Reflection

• PReP without Reflection retains a multi-step south-then-east plan and asks which step applies before selecting a route.In the example, it stays on Step 1 and does not update the plan.Source: §G.3 · p. 13

G.4 Prompt of PReP without Planning

• PReP without Planning receives the inferred southeast, more-southerly goal direction and recent trajectory before choosing between north and south.The example includes an unvisited south connection and a visited north connection.Source: §G.4 · p. 13

G.5 Prompt of CoT

• The CoT prompt reasons that Southwest is preferable to Northeast because it aligns with the goal’s southern component.The example frames both available moves as initially counterproductive for a goal inferred to be 51 steps south and 51 steps east.Source: §G.5 · p. 13

G.6 Prompt of InnerMonologue

• InnerMonologue prompts the agent to revise a southward plan when South is unavailable and choose Southwest as the closest available direction.The example sets a target at (0, -48) from (0, -27) and marks the prior action unsuccessful.Source: §G.6 · p. 14

G.7 Prompt of ProgPrompt; G.8 Prompt of CaP

• ProgPrompt and CaP examples encode movement as intersection-conditioned steps, then update their plans to choose an unvisited East route.ProgPrompt supplies a program-like route that switches directions at intersections; CaP uses conditional movement checks before its updated exploration plan.Source: §G.7 · p. 14, §G.8 · p. 14

G.9 Prompt of DEPS

• The DEPS prompt example begins with two visited neighboring routes and a planner instruction to move toward the nearest unvisited location.Source: §G.9 · p. 14
• The plan says to continue in the new direction while unvisited areas remain and movement brings the agent closer to the goal.Source: §G.9 · p. 14
• If the goal is not found and no further paths remain in the new direction, the agent should reassess its position.Source: §G.9 · p. 14
• After reassessing, the agent should plan further steps based on the remaining unvisited areas.Source: §G.9 · p. 14
• The prompt asks whether to update the plan and, when appropriate, requests a new plan.The example response sets “yes_or_no” to “yes.”Source: §G.9 · p. 14
• The prompt then asks the agent to choose its next action from the directions in the current connection.The example lists Southeast and Northwest as the available choices.Source: §G.9 · p. 14
• The example diagnoses a failed move because both available directions lead to previously visited locations.The agent reports being at (18, -8) and identifies Southeast and Northwest as the visited directions.Source: §G.9 · p. 14
• The revised plan directs the agent to the nearest unvisited area by choosing a direction not previously available.Source: §G.9 · p. 14