PaperlayerView exact revision on arXiv

Source-linked AI summary

Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning

Joan Perez

arXiv:2609.39437v1cs.LG

TL;DR

Geographic evaluations generally do not establish whether models use supplied context or override it with parametric recall. This paper builds an open network-based retrieval pipeline and claim-level benchmark, finding that resistance to planted false premises varies more by model family and generation than by scale, while brief-reading competence is partly separate.

  • Problem

    Geographic evaluations generally assess world-correctness rather than whether models use supplied context, leaving faithfulness to the brief unmeasured.

    Source: §1 · p. 1

  • Method

    An open pipeline retrieves data over a pedestrian-network catchment, computes a compact spatial brief, and evaluates claim source and correctness with planted false premises.

    Source: §1 · p. 1, §7 · p. 16

  • Results

    Resistance to false premises varies more by model family and generation than by size, while brief-reading competence is a partly separate capacity.

    Source: §7 · p. 16

  • Takeaways & Limitations

    Faithfulness is measurable as a distribution over claim-level labels across repeated generations, rather than as a single right-or-wrong answer.

    Source: §7 · p. 16

    The study covers three sites with one point each and sixteen configurations from three model lineages, so generalisation requires broader testing.

    Source: §6.4 · p. 15

Abstract

from arXiv · show

Large language models (LLMs) encode substantial latent geographic knowledge, yet they reason poorly over space and are unreliable when queried from coordinates alone. Useful behaviour emerges only when structured spatial context is supplied in the prompt. This raises a question geographic evaluation has left unexamined: once the right context is supplied, does the model reason from it, or override it with its own parametric recall? We take up this question with an open pipeline for network-based spatial context retriev-al. In it, the surroundings of a selected point are defined by the pedestrian street network, the area actually reachable on foot. Using only open data and open-weight models, the pipeline retrieves features from OpenStreetMap and the GHS-POP population grid, computes indicators over the network catchment in code, and injects them as a compact spatial brief. On this basis we build a faithfulness benchmark. It labels every claim a model makes by its source (grounded in the brief, or drawn from training knowledge) and its correctness, and it probes each case with a planted false premise that the brief refutes. We evaluate sixteen open-weight model configurations across three families (Qwen, Gemma and Llama, with Gemma in two generations), four size classes and, where available, both thinking and non-thinking modes, on three con-trasting cities, resampling every case over ten seeds. The results show that resistance to the planted premise varies more strongly by model family and generation than by scale, while brief-reading competence forms a partly separate dimension. These behaviours are not captured by conventional world-correctness scores or single-shot evaluation. We release the implementation, spatial briefs, model outputs, and claim-level labels as a reproducible workflow at github.com/perezjoan/NSCR-LLM.

1. Introduction

Geographic knowledge in LLMs does not ensure reliable spatial reasoning, and geographic evaluations have largely left untested whether models use supplied context faithfully. The paper presents an open pipeline and claim-level benchmark for testing that question across three cases and sixteen model configurations.

Motivation

  • Existing geographic evaluations generally cannot tell whether a model reasons from supplied facts or overrides them with parametric recall.Faithfulness to context is distinct from correctness about the world.Source: §1 · p. 1

Approach

  • The pipeline defines a location’s surroundings by pedestrian-network reachability and computes open-data indicators into a structured brief for the model.The system uses OpenStreetMap and GHS-POP, with no fine-tuning or proprietary services.Source: §1 · pp. 1–2

Evaluation

  • Sixteen open-weight configurations are tested across three contrasting city cases, with each case resampled over ten seeds.The evaluation examines brief use, resistance to a planted false premise, and variation across models and cases.Source: §1 · p. 2

2. Motivation and related work

Prior work finds useful geographic knowledge can be surfaced with context, while spatial reasoning remains unreliable and context faithfulness is not guaranteed. This section motivates a network-based retrieval approach and claim-level evaluation with planted false premises.

2.1. Language models meet geography

  • Models can surface latent geographic knowledge with contextual prompts, yet struggle with spatial operations such as distance and direction.Research also explores tool-using GIS systems and retrieval-augmented approaches, including ChatMap’s point-centered OSM prompts and fine-tuning.Source: §2.1 · pp. 2–3

2.2. Faithfulness to supplied context

  • Models may ignore supplied context, persist with conflicting priors, or accept false external evidence, so retrieval alone does not guarantee faithful use.The cited studies describe conflict between retrieved context and internal priors.Source: §2.2 · p. 3
  • World-correctness scores cannot distinguish grounded answers from answers that ignore the brief, motivating claim-level labels for source and correctness.The benchmark also plants a false premise to test whether models defend the brief or accept the assertion.Source: §2.2 · p. 3, §2.3 · p. 3
  • Repeated evaluation across seeds measures faithfulness and false-premise resistance as tendencies rather than properties of one generation.The paper notes that stochastic models can vary in their adherence to context across samples.Source: §2.2 · p. 3

2.3. The gap and our approach

  • For reachable-area questions, straight-line distance may misrepresent accessibility, so the approach defines context with a network catchment and pre-computes indicators.The resulting brief separates deterministic spatial computation from model interpretation.Source: §2.3 · p. 3, Figure 1 · p. 4

3. Method

The method separates deterministic spatial retrieval and computation from model interpretation, then measures faithfulness by labeling individual claims and testing responses to planted false premises. It uses a reproducible open-data and open-weight workflow.

3.1. Overview and design principles

  • A two-stage system turns a map click into a conversation: code builds a spatial brief, which is injected into the model’s context for questioning.The design separates computation from interpretation, uses a pedestrian-network extent, and relies on open data and open-weight models without fine-tuning.Source: §3.1 · p. 4

3.2. The network catchment

  • The catchment follows shortest-path walking distance along street segments, then buffers and fills the reachable network using a recorded block-depth parameter.Segments are cut at the exact remaining distance when only part falls within the walking budget; the filled geometry is the shared aggregation window.Source: §3.2 · pp. 4–5

3.3. Retrieving and aggregating the layers

  • OpenStreetMap layers and GHS-POP residential population are retrieved within the catchment, with filtering and aggregation rules addressing mapped geometry and data properties.Population cells are areal-weighted; the brief notes that modelled residential counts can be low in busy commercial or mixed-use areas.Source: §3.3 · p. 5

3.4. The spatial brief

  • Code reduces retrieved layers to named indicators in a structured brief, with ratios formed from aggregated totals and a fixed glossary explaining each field.The brief contains finished values, so the model does no arithmetic; deterministic aggregation replaces semantic-similarity retrieval.Source: §3.4 · p. 5

3.5. Injecting the brief into an open-weight model

  • The brief, glossary, and persona form the prompt; sixteen configurations from eleven checkpoints vary by family, size, and available thinking mode.Configurations use four-bit quantisation and manufacturer-recommended sampling settings, which differ between families.Source: §3.5 · p. 6, Table 1 · p. 6

3.6. A faithfulness benchmark

  • Answers are segmented into atomic claims and labeled independently for source and correctness, distinguishing brief-true, brief-false, recall, and hallucination.Profiles aggregate grounding share, errors among brief-sourced claims, and the split between recall and hallucination.Source: §3.6 · p. 7

3.7. Implementation and reproducibility

  • A notebook implements both pipeline stages and batch-runs fixed question sequences over repeated seeds; released materials include briefs, outputs, and claim labels.Frozen briefs preserve the evaluated records so results remain reproducible despite later upstream-data changes.Source: §3.7 · pp. 7–8

4. Application

The application tests grounded reasoning across three contrasting urban cases, using a repeated claim-level benchmark and calibrating the language-model judge against human labels.

4.1. Three case studies

4.2. Benchmark protocol

  • The protocol holds four-bit quantization and the brief, persona, and glossary fixed; three memory-linked questions are repeated across ten seeds, yielding 1,434 retained responses.Two Paris sequences for Llama-3.2-1B were excluded after reaching the generation limit.Source: §4.2 · p. 10
  • Atomic claims receive source and correctness labels, while each planted-premise response scores 0–1 per seed according to rejection with brief evidence, hedging, or acceptance.The ten-seed trap-resistance score ranges from 0 to 10 per case.Source: §4.2 · p. 10

4.3. Calibrating the language-model judge

  • 85.0% pooled agreement, with Cohen’s κ=0.685 (95% CI 0.66–0.71) and Krippendorff’s α=0.684; city-level κ ranges from 0.62 in Paris to 0.77 in Hanoi.Agreement is higher for retrieval and reasoning (κ=0.72 and 0.68) than for the trap (κ=0.62).Source: §4.3 · p. 10, Table 2 · p. 11
  • Judge disagreements clustered around brief-grounded inference versus recall, trap endorsement, and error propagation; the judge assigned the more severe label in 65% of disagreements.The boundary between brief-grounded inference and recall accounted for 88 disagreements.Source: §4.3 · p. 11
  • Per-configuration error rates correlated highly between annotators (Spearman ρ=0.96 pooled), but the judge’s stricter convention qualifies interpretation of absolute error rates.The released seed-1 annotations and agreement analysis provide a human comparison baseline.Source: §4.3 · p. 11

5. Results

Trap resistance varies more by model family and generation than by scale, while brief-reading skill forms a partly separate dimension. Outcomes also differ across cities and seeds.

5.1. Trap resistance is gated by model family, not by size

  • Gemma-4 scores 22.5–23 of 30 on Q3 versus Qwen3 at 10–20.5, Gemma-3 at 1–4, and Llama at 2–6.5; family gaps outweigh size.At 12B, Gemma-4 scores six times Gemma-3; Qwen3-1.7B without thinking outscores every Gemma-3 and Llama configuration.Source: §5.1 · p. 11

5.2. Trap failure surfaces as hallucination, or as misreading

  • In Chicago, trap-resisting Qwen3 and Gemma-4 keep hallucination at 0%–2.1%, versus 9%–16% for Gemma-3 and Llama; Hanoi failures mostly appear as brief misreadings.Hanoi Q3 brief-false shares reach 64% for Gemma-3-4B.Source: §5.2 · pp. 11–12

5.3. The smallest models fail on a second, independent axis

  • Gemma-3-1B and Llama-3.2-1B have only 35–44% brief-true shares, with field-binding errors, fabrication, list-dumping, and occasional nontermination.Qwen3-1.7B stays within the brief at 56–84% brief-true, showing that small size alone does not produce these failures.Source: §5.3 · pp. 12–13

5.4. Difficulty rises from Chicago to Paris to Hanoi

  • Trap resistance falls from 65% of the maximum in Chicago to 43% in Paris and 14% in Hanoi as refutation demands increasingly complex interpretation.Only Qwen3-14B without thinking reliably resists the Hanoi premise, scoring 8.5 of 10.Source: §5.4 · p. 14

5.5. Thinking mode buys latency, not grounding

  • Thinking variants take 2.6 to 4.6 times longer, while brief-true shares shift by at most 2.5 points and trap-resistance effects vary by size.Trap resistance improves at 1.7B and 4B but falls at 14B; thinking does not make outputs more consistent.Source: §5.5 · p. 14

5.6. Run-to-run consistency

  • Composite instability falls from about 9 percentage points at 1B to 4 from 8B upwards, while family averages range from Gemma-4 at 3.1 to Llama at 8.0.Figure 4 profiles grounding, hallucination, trap resistance, stability, concision, and speed; one-shot results can capture either of a model’s varying behaviors.Source: §5 · p. 11, Figure 4 · p. 13, §5.6 · p. 14

6. Discussion

Faithfulness comprises distinct capacities—reading a supplied brief, resisting a false premise, and doing so consistently—and these do not scale together. The discussion draws implications for model choice, richer context, streetscape data, study limitations, and geographic reasoning beyond this benchmark.

6.1. Implications for model choice

  • Trap resistance tracks model family and generation more than parameter count, while reading accuracy does not predict whether models defend the brief.Thinking mode increases response time without consistently improving grounding or resistance, and trap outcomes can flip across seeds.Source: §6.1 · p. 14

6.2. A richer brief

  • A proximity matrix or local spatial statistics could add network accessibility and within-catchment variation to the compact scalar brief.The Chicago food-desert signal could become a measured distance, while local statistics could distinguish uniform density from a gradient.Source: §6.2 · p. 15
  • Enrichment is most useful for models that already read and defend a compact brief, because added fields also enlarge the surface for misreading.In Hanoi, a caveat about resident counts was turned into support for the user's false premise.Source: §6.2 · p. 15

6.3. Coupling to streetscape analysis

  • The brief and streetscape scoring could be coupled in either direction, but perceptual scores remain model estimates and warrant more caution than counted quantities.SAGAI scene indicators could enter the brief, or the brief could inform the vision-language model's scene scores.Source: §6.3 · p. 15

6.4. Limitations

  • Open-data indicators have distinct vulnerabilities: building measures depend on variable OpenStreetMap tagging, while GHS-POP residential counts can understate activity in mixed-use districts.The Hanoi trap exploits the population-grid limitation, which belongs to the data rather than to a model.Source: §6.4 · p. 15
  • Claim-label uncertainty is greater for trap questions and for separating brief-grounded inference from recall than for overall error rates.A language-model judge was calibrated against a human annotator on one seed per configuration.Source: §6.4 · p. 15
  • Generalization is bounded by three sites with one point each and sixteen configurations from three lineages; results also use four-bit quantisation and one sampling configuration per model.More sites, model families, and checks at full precision or other temperatures could test whether family- or scale-linked differences change.Source: §6.4 · p. 15

6.5. Beyond geography

  • Claim-source labelling and planted-premise tests could transfer beyond geography where ground truth is reproducible, but geographic priors are local, sometimes outdated, and easily mistaken for facts about a specific point.The geographic setup computes ground truth deterministically from a network catchment over open data.Source: §6.5 · p. 16

7. Conclusion

The paper presents an open pipeline and claim-level benchmark for testing whether models reason from supplied geographic context. Its results separate reading competence from resistance to false premises, with resistance varying more by family and generation than by size.

7. Conclusion

  • The pipeline computes a network catchment and compact brief from open data, then labels model claims by source and correctness and tests each case with a planted false premise.The benchmark covers three cities, eleven open-weight models in sixteen configurations, and ten resampled seeds.Source: §7 · p. 16
  • Resistance to false premises varies more by model family and generation than by size, while reading competence is a separate axis and trap outcomes can flip across seeds.Gemma-4 and most Qwen3 configurations defend the brief, while Gemma-3 and Llama accept the premise almost everywhere.Source: §7 · p. 16
  • Faithfulness is measurable as a distribution over claims when evaluation labels source and correctness and resamples generations, rather than treating one answer as a verdict.For grounded decision-support uses, model choice is not determined by scale or reasoning mode; thinking mode buys latency more reliably than grounding.Source: §7 · p. 16
  • The findings are bounded by three cities with one catchment each, sixteen configurations, one planted premise per case, four-bit quantisation, and one sampling configuration per model.Source: §7 · p. 16

A. Indicators computed over the catchment

The appendix describes indicator computation over the filled network catchment and provides a table of indicator definitions and computation rules.

A. Indicators computed over the catchment

  • Indicators are computed in code over the filled network catchment, so the model performs no arithmetic.Totals are aggregated first, and ratios are then formed from those aggregated totals.Source: A. Indicators computed over the… · p. 18
  • Table A1 lists the spatial-brief indicators alongside each indicator's definition and computation rule.Source: Table A1 · p. 18

B. Glossary supplied to the model

The supplied glossary explains the fields in the spatial brief, including how the area is defined and what population and built-environment indicators represent.

B. Glossary supplied to the model

  • The glossary is injected verbatim between the persona and the brief in every prompt.Source: B. Glossary supplied to the model · p. 18
  • The glossary defines retrieval method, location, population, roads, points of interest, and pre-computed indicators used in the brief.It distinguishes residential GHS-POP counts from daytime workers or visitors and explains that commercial or mixed-use areas may therefore have low population figures despite being busy.Source: B. Glossary supplied to the model · p. 18

C. Full results by model and case

Table C1 reports claim composition, Q3 trap resistance, seed instability, response length and response time by model configuration and case, with a final block pooling the three cases.

C. Full results by model and case

  • Table C1 covers claim composition, Q3 trap resistance, seed instability, response length and response time by model configuration and case, then pools the three cases in its final block.The table provides the claim counts behind Figure 3 and the measures used in Figure 4.Source: C. Full results by model and case · p. 19, Table C1 · p. 19
  • The table distinguishes claim-source and correctness shares from Q3 trap resistance and seed instability; pooled response length and time are response-weighted means.B denotes parameters in billions, T thinking mode, C labelled claims, E ECHO lines; TR is Q3 trap resistance, SI seed instability, AWC average words and ART average response time.Source: C. Full results by model and case · p. 20
  • For Paris Llama-3.2-1B, two seeds were excluded after Q1 hit the 10,000-token limit in a degenerate loop, leaving trap resistance out of 8.The pooled trap-resistance score uses 28 seeds for this model.Source: C. Full results by model and case · p. 20

Related papers

Loading 2609.39437v1…