Source-linked AI summary
Structured Maps for Landmark-Based Route Instructions with LLMs
Mifu Matsumoto, Yuki Nakanishi, Tomoya Hashiguchi, Hiroaki Ohshima
TL;DR
LLMs need to reason about map geometry and human landmark preferences, but the best map representation for them is unsettled and image inputs challenge fine-grained spatial reading. The paper encodes local maps as structured SVG text and evaluates two landmark-selection tasks. SVG matches or outperforms PNG inputs, and fine-tuning improves local models in the reported evaluation.
Problem
The best way to structure map data for LLM landmark selection is unsettled, while image inputs make fine-grained label and spatial-relation reading challenging.
Source: §1 · p. 1Method
The paper represents local map geometry and semantics as structured SVG text and evaluates it on turn and destination selection using human-written descriptions linked to OpenStreetMap.
Source: §1 · p. 2Results
Structured SVG consistently matched or outperformed PNG, while parameter-efficient fine-tuning yielded substantial performance gains in local open-weights models.
Source: §1 · p. 2, §8 · p. 14Takeaways & Limitations
The findings identify structured SVG as an effective map interface for LLM landmark selection and fine-tuning as a pathway for aligning local models with human spatial communication.
Source: §8 · p. 14The study uses exact-match accuracy without human evaluation of linguistic naturalness or practical clarity, omits visual-salience features, and has a modest dataset.
Source: §7.4 · p. 13
Abstract
from arXiv · showhide
To generate landmark-based route instructions, models must understand both map geometry and human landmark preferences. Although large language models (LLMs) show great abilities in language tasks, the best way to structure map data for these models is not yet established. This paper proposes a structured text representation for landmark selection tasks in route navigation, utilizing SVG format to encode spatial features. The structured text maintains a geometric connection with rendered map images, allowing language models to directly access spatial and semantic data. Using a new dataset created from human-written route descriptions and OpenStreetMap data, we evaluate models on two tasks: landmark selection at turns and around destinations. Experiments with five LLMs show that the proposed structured text input performs as well as or better than image-based PNG inputs. In addition, task-specific fine-tuning enables smaller local LLMs to effectively leverage map layouts to achieve high selection accuracy. These results demonstrate that structured text is an efficient and lightweight interface for spatial reasoning in human-oriented navigation.
1 Introduction
The paper studies how LLMs can select human-preferred landmarks from local maps, where raw-image map reading makes fine-grained spatial and label interpretation challenging. It proposes structured SVG map text and evaluates it alongside PNG inputs on two landmark-selection tasks.
1 Introduction
- People often find distance- and street-name-based instructions difficult to follow because distances are hard to estimate and abstract road names are hard to locate.Landmark-based instructions instead connect navigation actions to visible features.Source: §1 · p. 1
- SVG encodes coordinates and semantic categories as text while retaining a geometric connection to the rendered map, enabling a controlled comparison with PNG.Image-based inputs require models to infer spatial relations from pixels, including small labels and relative positions.Source: §1 · p. 2
- The evaluation defines turn and destination landmark-selection tasks using human-written route descriptions linked to OpenStreetMap objects.The comparison asks which representation is more effective and whether SVG-based fine-tuning improves smaller local LLMs.Source: §1 · p. 2
- Structured SVG text performs as well as or better than image-based PNG inputs for landmark selection.The paper also investigates whether task-specific fine-tuning helps smaller local LLMs use map layouts and learn human landmark preferences.Source: §1 · pp. 2–3
2 Related Work
Landmark choice must balance spatial relevance, visual salience, and communicative utility. This paper compares direct selection from structured maps with prior approaches to route description and map reasoning.
2.1 Landmarks in Spatial Cognition and Landmark Selection
- An optimal landmark balances spatial relevance, visual salience, and communicative utility, so the nearest object is not always sufficient.The paper frames accurate imitation of human landmark selection as a challenge for automation.Source: §2.1 · p. 3
2.2 Route and Destination Description Generation
- Prior route-generation methods include rule-based templates, graph-to-text approaches, and destination-description pipelines that pre-filter landmarks using geometric scoring.This work instead selects directly from raw structured map inputs and tests whether fine-tuning on human data helps models learn preferences without manual scoring rules.Source: §2.2 · p. 3
2.3 LLMs and VLMs for Map Understanding
- VLMs can struggle with fine-grained map reading, text labels, and route tracing, while image encoders may abstract away precise spatial relations.Text-based LLMs also have limitations maintaining accurate internal 2D spatial representations.Source: §2.3 · p. 4
- Prior work finds structured text useful for reasoning in robotics, web, chart, and graph tasks; this study directly compares SVG text with rendered PNG maps for landmark selection.The comparison extends the observation that textual graph representations can outperform visual graph images on some reasoning tasks.Source: §2.3 · p. 4
3 Task Definition
The paper defines landmark selection as matching a human route-description choice given a local map and navigation context. It distinguishes turn and destination tasks, both requiring spatial, semantic, and human-oriented reasoning.
3 Task Definition
- Across both tasks, the model selects a landmark from a local map and navigation context, and accuracy is based on an exact match to the human-used landmark.The tasks require reasoning about both spatial coordinates and semantic name attributes.Source: §3 · p. 4, §3.1 · p. 5
3.1 Task Settings
- The task setting assumes the local navigation context is already given; it does not find a complete route from a start to a destination.The model receives candidate landmarks and must output the correct landmark identifier.Source: §3.1 · pp. 4–5
3.2 Task 1: Landmark Selection at Turns
- For turn selection, the model receives an intersection map and turning action, then selects the landmark that best represents that turn.Candidates include names, semantic categories, and coordinates; evaluation checks whether the selected name matches the human choice.Source: §3.2 · p. 5
3.3 Task 2: Landmark Selection around Destinations
- For destination selection, the model chooses a nearby landmark that best describes the destination's relative location.The task uses the relations “next to,” “across from,” and “beside,” with a destination-centered map and named, categorized, positioned candidates.Source: §3.3 · pp. 5–6
3.4 Characteristics of the Tasks
- The two tasks require interpreting map geometry, choosing landmarks useful for human communication, and grounding candidates by their proper names.The geometrically closest landmark may be obscure or hidden from the user's perspective.Source: §3.4 · p. 6
4 Dataset Construction
The dataset links human-written restaurant access descriptions to verified OpenStreetMap landmarks for two tasks: landmark selection at turns and around destinations. Filtering and manual map checks produced 500 turn instances and 339 destination instances.
4.1 Source of Human-Written Route Descriptions
- Restaurant access descriptions provided human-written examples of practical landmarks, filtered from an initial pool of 136,490 descriptions.Descriptions limited to travel time or distance without explicit landmarks were excluded.Source: §4.1 · p. 6, §4 · p. 6
4.2 Filtering and Extraction via LLMs
- Keyword screening and GPT-3.5 Turbo verification identified descriptions with concrete landmarks and extracted their names.Turn-task screening sought explicit left- or right-turn expressions; destination-task screening used three target keywords.Source: §4.2 · p. 7
4.3 Linking to OpenStreetMap and Manual Verification
- Researchers linked extracted landmark names to nearby OSM objects, then manually corrected mismatches and verified each task’s spatial context.Candidate objects were queried within 1,000 meters of each restaurant; geometry and category tags supplied spatial attributes.Source: §4.3 · p. 7
- The verified datasets contain 500 instances for Task 1 and 339 for Task 2.Multiple turning points in one description were treated as independent instances, while descriptions that could not be verified on the map were discarded.Source: §4.3 · p. 7
5 Structured Map Representation
The paper represents local maps as structured SVG text, preserving spatial relations while exposing names, coordinates, and semantic categories directly to LLMs. The same underlying map data supports a controlled comparison with rendered PNG inputs.
5 Structured Map Representation
- SVG text exposes map names and categories alongside geometric positions, preserving spatial relations and matching the geometry of a rendered map.The text and image modalities use identical underlying map data, enabling a controlled comparison.Source: §5 · p. 8
5.1 Local Map Extraction
- Each local map centers on the target intersection or destination and covers approximately 124 × 124 meters.The extraction uses zoom level 18 and a 256×256-pixel rendering size.Source: §5.1 · p. 8
5.2 Coordinate Normalization
- Map coordinates scale to a 100 × 100 system with the target fixed at (50, 50), making relative positions easier to infer from text.For example, a landmark at (51, 32) lies slightly right and above the center.Source: §5.2 · p. 8, Figure 3 · p. 9
5.3 SVG Text Components and Structure
- The SVG encodes roads as line segments, route vectors, a target point, and landmark groups with name, category, and coordinate attributes.The route vectors indicate how a user enters and leaves an intersection in Task 1.Source: §5.3 · pp. 9–10
5.4 Input Modalities and Prompting Setup
- The model receives structured SVG text with a task-specific prompt and returns a JSON name field matching a map attribute.PNG inputs use equivalent task instructions and JSON output rules, with the map rendered from the same geographic data.Source: §5.4 · p. 10
6 Experimental Setup
The experiments compare PNG and SVG inputs on identical map information, then evaluate task-specific adaptation using SVG for local models. Landmark selection accuracy is measured against human references, with linguistic phrasing excluded from scoring.
6.1 Experimental Configuration
- The first stage compares PNG and SVG modalities using identical local-map information and equivalent task descriptions and output rules.The second stage evaluates local models before and after task-specific adaptation using SVG input across both tasks.Source: §6.1 · p. 10
6.2 Fine-Tuning Environment and Hyperparameters
- Fine-tuning uses LoRA or QLoRA with rank 16, alpha 16, a 2 × 10^-5 learning rate, and validation-loss early stopping.The AdamW 8-bit optimizer and a constant learning-rate schedule are used.Source: §6.2 · p. 11, Table 1 · p. 11
6.3 Evaluation Metric and Output Parsing
- Selection accuracy scores exact matches between predicted and human-reference landmark names; invalid JSON or missing name fields count as incorrect.The evaluation isolates spatial reasoning by not scoring the generated instruction’s wording with lexical similarity metrics.Source: §6.3 · p. 11
7 Results and Discussion
Across the experiments, structured SVG map input matched or outperformed PNG, and task-specific fine-tuning generally improved local models. Error analysis identified remaining mismatches with human choices, while the authors note evaluation, representation, and dataset boundaries.
7.1 RQ1: Effect of Map Representation
- Across configurations, SVG matched or beat PNG; Gemma3:27B rose from 14% to 27% in Task 1, while GPT-5.2 reached 55% in Task 2 with SVG.The authors attribute the advantage to exact names and coordinates and an explicit spatial layout, rather than requiring models to infer relations from pixels.Source: Table 2 · p. 12, §7.1 · p. 12
7.2 RQ2: Effect of Fine-Tuning
- Task-specific LoRA or QLoRA adaptation generally boosted local models, though Gemma3:27B dipped on destination selection.In Task 1, Llama-3-ELYZA-JP-8B reached 49% and Gemma3:27B reached 53%; in Task 2, ELYZA-Thinking-1.0-Qwen-32B rose to 54%, while Gemma3:27B changed from 39% to 36%.Source: Table 3 · p. 12, §7.2 · pp. 12–13
7.3 Qualitative Error Analysis
- Fine-tuned models missed human references through plausible alternative choices, incorrect side-of-street alignment, or rare invalid JSON output.The authors note that multiple salient landmarks in dense urban layouts can make a single human reference non-unique.Source: §7.3 · p. 13
7.4 Limitations and Future Directions
- The evaluation uses exact-match accuracy without human assessments, omits visual-salience features, and relies on a modest dataset.The authors call for multi-reference or field-study evaluation and larger datasets across diverse environments.Source: §7.4 · p. 13
8 Conclusion
The paper presents SVG text maps as an input for LLM-based landmark selection and reports that they matched or outperformed PNG inputs. It also reports gains from parameter-efficient fine-tuning, while identifying evaluation, visual-salience, and dataset boundaries.
8 Conclusion
- Structured SVG representation consistently matched or outperformed visual PNG inputs in the reported evaluations.The representation converts local geographic data into structured text for LLMs.Source: §8 · p. 14
- Parameter-efficient fine-tuning on a modest dataset substantially improved performance in local open-weights models.The authors describe this as an efficient pathway for alignment with human spatial communication.Source: §8 · p. 14
- Future work should test landmark utility with multi-reference metrics or human assessments and add visual-salience metadata such as building colors, dimensions, and heights.The paper also identifies the need to expand the dataset across diverse spatial environments.Source: §8 · p. 14
A Experimental Prompts
The appendix provides English translations of prompts administered in Japanese. The templates specify distinct instructions for turn and destination landmark tasks, along with SVG data conventions and expected outputs.
A Experimental Prompts
- The prompt templates require structured JSON output, including the selected landmark name and an instruction.The appendix says the displayed English templates reflect the semantic structure of prompts administered in Japanese.Source: A Experimental Prompts · pp. 16–17
- For turn guidance, the prompt asks the model to use movement direction and SVG map data to select a landmark and produce a navigation instruction.The analysis steps direct it to check the start-to-goal route angle, inspect POI coordinates relative to the intersection and roads, and generate a user-centric guide.Source: A Experimental Prompts · p. 17
- For destination guidance, the prompt asks for a visible landmark and a natural description of its spatial relation to the destination.It cautions against selecting only the nearest POI and directs the model to consider road layout and visibility inferred from data category.Source: A Experimental Prompts · p. 18
- The SVG schema places the target and up to 16 nearby POIs in a 0–100 coordinate system, with road lines representing the layout.POI attributes include names and categories; the turn-task schema also specifies turn direction and start and goal routes.Source: A Experimental Prompts · pp. 17–18