Source-linked AI summary
The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization
Ruixing Zhang, Zihan Liu, Leilei Sun, Tongyu Zhu, Weifeng Lv
TL;DR
Audio geo-localization lacks a systematic benchmark and a quantitative way to identify geographically informative recordings. AGL1K combines a curated benchmark and an Audio Localizability metric with evaluation of 16 ALMs, which exhibit emerging capability while the task remains challenging.
Problem
Audio geo-localization lacks a systematic benchmark and a quantitative measure for identifying recordings with meaningful geographic signals.
Source: §1 · p. 1Method
AGL1K curates crowd-sourced audio-location pairs using Audio Localizability, which aggregates positive and negative sound-category evidence to estimate each recording’s geographic informativeness.
Source: §1 · p. 1, §3.3.1 · p. 3Results
Takeaways & Limitations
The error analysis points to fine-grained perception, regional bias, and compositional reasoning as directions for improving audio models.
Source: §1 · p. 2, §5 · p. 8Sample counts are uneven across continents, with Europe, Asia, and North America overrepresented relative to Africa, Oceania, and South America.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Geo-localization aims to infer the geographic origin of a given signal. In computer vision, geo-localization has served as a demanding benchmark for compositional reasoning and is relevant to public safety. In contrast, progress on audio geo-localization has been constrained by the lack of high-quality audio-location pairs. To address this gap, we introduce AGL1K, the first audio geo-localization benchmark for audio language models (ALMs), spanning 72 countries and territories. To extract reliably localizable samples from a crowd-sourced platform, we propose the Audio Localizability metric that quantifies the informativeness of each recording, yielding 1,444 curated audio clips. Evaluations on 16 ALMs show that ALMs have emerged with audio geo-localization capability. We find that closed-source models substantially outperform open-source models, and that linguistic clues often dominate as a scaffold for prediction. We further analyze ALMs' reasoning traces, regional bias, error causes, and the interpretability of the localizability metric. Overall, AGL1K establishes a benchmark for audio geo-localization and may advance ALMs with better geospatial reasoning capability.
1 Introduction
Audio geo-localization lacks a systematic benchmark and a quantitative way to identify informative recordings. AGL1K addresses these gaps, and evaluation shows emerging model capability alongside persistent challenge.
1 Introduction
- Across 16 ALMs, geo-localization ability has emerged, but AGL1K remains challenging and closed-source models outperform open-source models.Leading models can over-commit to a single clue, while open-source models show fine-grained perception limitations.Source: §1 · p. 2
- The identified improvement directions are finer-grained perception, reduced regional bias, and stronger compositional reasoning across multiple clues.Open-source models often misidentify languages and subtle acoustic clues, while prediction imbalances persist across models.Source: §1 · p. 2
- AGL1K introduces a benchmark of 1,444 audio clips from 72 countries, spanning diverse acoustic scenes.The benchmark is designed to assess compositional reasoning in audio-language models.Source: §1 · pp. 1–2
- Audio Localizability estimates each recording’s geographic informativeness by aggregating evidence from positive and negative sound categories.The metric is used to filter for localizable recordings.Source: §1 · pp. 1–2
2 Related Work
Geo-localization research has largely focused on images, while audio geo-localization remains comparatively underexplored. Existing ALM benchmarks cover several tasks, but none is dedicated to audio geo-localization.
2.1 Geo-Localization
- Image geo-localization uses classification, retrieval, and vision-language approaches, each with distinct scalability or data requirements.Fine-grained classification needs many classes, while retrieval depends on large annotated image databases.Source: §2.1 · pp. 2–3
- Audio geo-localization remains underexplored, with existing benchmarks often restricted to narrow domains such as bird vocalizations.The cited work contrasts this with broader image geo-localization research.Source: §2.1 · p. 3
2.2 Audio-Language Models
- ALM benchmarks assess tasks from transcription and vocal-sound classification to spatial reasoning and causal discovery, but none targets audio geo-localization.Source: §2.2 · p. 3
3 Audio Geo-Localization Benchmark
AGL1K is built from crowd-sourced audio, screened for recording quality and geographic informativeness, then manually curated. Its localizability measure and examples characterize which sound categories contribute useful geographic clues.
3 Audio Geo-Localization Benchmark
- The benchmark pipeline acquires audio-location pairs, applies coarse quality control, selects localizable recordings, and post-processes ALM responses.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Dataset Acquisition
- Aporee provides tens of thousands of crowd-sourced, geo-tagged audio-location pairs released under an Apache license.Source: §3.1 · p. 3
3.2 Initial Filtering
- Four acoustic filters remove low-energy, noise-like, heavily clipped, or overly monotonous recordings while retaining clips with acoustic structure.The filters use RMS Energy, Spectral Flatness, Clipping Ratio, and Acoustic Complexity.Source: §3.2 · p. 3
3.3.1 Localizability Calculation
- Localizability models a recording’s net positive versus negative sound-category contribution, with informative categories expected to improve geographic inference and uninformative content to obscure clues.Category presence is measured as a fraction of recording time, and category contribution strengths are estimated from model reasoning and judgments.Source: §3.3.1 · pp. 3–4
- Categories are assigned positive or negative roles from judged contribution patterns, and recordings above the localizability threshold are manually curated into 1,444 clips balanced for speech presence.The stated settings are α = 1/3, γ = 1000 km, and θ = 1.Source: §3.3.1 · p. 4
3.3.2 Top Positive and Negative Categories
- Speech is the strongest positive category; regionally distributed rail transport and waves also score highly, whereas ubiquitous sounds such as rain and wood contribute negatively.The category attributions are reported as aligning with human intuition.Source: Figure 3 · p. 4, §3.3.2 · p. 5
3.3.3 Localizability Examples
- Thunder and rain yield the lowest example scores, while birdsong can raise localizability; masking foreground noise weakens speech’s geographic informativeness.The figure presents example clips with localizability increasing from right to left.Source: §3.3.3 · p. 5, Figure 4 · p. 5
3.4 Post Processing
- Regional name variants are mapped to canonical labels for evaluation, and an interactive website supports global exploration of the task.Source: §3.4 · p. 5
4 Experiments
The experiments benchmark 16 audio-language models on audio geo-localization and examine their reasoning, regional prediction inequality, and error patterns. Closed-source models perform strongly relative to open-source models, while linguistic clues and regional imbalances shape results.
4.1.1 Benchmarking Models
- The evaluation benchmarks eight closed-source and eight open-source models using distance, geoscore, hierarchical and thresholded accuracy, reject rate, and speech-versus-non-speech error metrics.Open-source models other than Qwen3-Omni were deployed on an RTX 4090; other models were accessed through APIs with default settings.Source: §4.1.1 · pp. 5–6, §4.1.2 · p. 6
4.2 Comparison Results (RQ1)
- 0.82 continent-level accuracy and 0.51 country-level accuracy are achieved by Gemini 3 Pro, while the best open-source model, Mimo-audio, records an average error of 4853 km—more than twice Gemini 3 Pro’s.Table 1 reports model performance; the prose identifies Gemini 3 Pro’s leading accuracies and the gap to Mimo-audio.Source: Table 1 · p. 6, §4.2 · p. 6
- Speech substantially improves Gemini 3 Pro’s localization: its average error is 1548 km with speech, versus 3727 km for non-speech audio.Similar speech/non-speech gaps appear across all evaluated models, while purely environmental sounds remain more challenging.Source: §4.2 · p. 6
4.3 Benchmark Examples (RQ2)
- In three examples, Gemini 3 Pro combines linguistic, environmental, or contextual clues; other models sometimes overemphasize a clue or make unsupported guesses.Examples include an Essaouira recording where other models overemphasize Adhan, transit clues identifying Stockholm, and nonverbal clues identifying Berlin.Source: Figure 5 · p. 7, §4.3 · p. 7
4.4 Continent-Level Prediction Inequality (RQ3)
- Continent-level prediction inequality persists: Qwen3-Omni often predicts North America for other continents, while Oceania is particularly unstable for GPT-4o Audio.Gemini 3 Pro has the strongest diagonal, especially for Africa, Asia, and Europe, with weaker performance on Oceania and South America.Source: §4.4 · p. 7, Figure 6 · p. 8
4.5 Error Distribution (RQ4)
- Error profiles differ by model: language ambiguity dominates Gemini 3 Pro’s errors, educated guesses affect GPT-4o Audio, and label misidentification is most common for Mimo-Audio.The analysis annotates 300 country-level misclassifications across three representative models; the authors recommend stronger multilingual perception, evidence-aware reasoning, and integrating complementary clues.Source: §4.5 · p. 8
5 Conclusion
The paper introduces AGL1K, a curated benchmark for evaluating audio geo-localization in audio-language models. Evaluation and error analyses show emerging capability alongside persistent challenges, motivating improvements in perception, regional bias, and compositional reasoning.
5 Conclusion
- AGL1K contains 1,444 user-uploaded audio clips from 72 countries and territories across six continents, filtered by Audio Localizability and spanning diverse acoustic scenes.Evaluation of 16 models finds emerging geo-localization ability while the benchmark remains challenging.Source: §5 · p. 8
- The analyses identify fine-grained perception, regional bias, and compositional reasoning as directions for improving audio-language models.The paper expects audio geo-localization to become an important benchmark for evaluating ALMs’ reasoning capability.Source: §5 · pp. 8–9
6 Limitations
The limitations section identifies uneven geographic representation and constraints tied to annotation-model choices and collected data. The accompanying appendix describes the source platform, an interactive human-localization interface, acoustic filters, evaluation metrics, and sound labels.
6 Limitations
- The benchmark’s continent representation is uneven: Europe, Asia, and North America are significantly overrepresented relative to Africa, Oceania, and South America.The authors attribute the imbalance to the distribution of the original data.Source: §6 · p. 9
- Sound-label diversity and the distribution of positive and negative attribution results are partly constrained by annotation-model choices and collected data.Source: §6 · p. 9
A.1 Introduction of Aporee Platform
- Aporee supplies user-uploaded, GPS-associated environmental recordings; the interactive platform asks participants to locate a clip on a map and logs anonymized predictions.The platform provides a reference baseline for comparison with model performance.Source: §A.1 · p. 11, §A.2 · p. 11, Figure 8 · p. 11
A.3 Formulation of the four acoustic filters
- The four acoustic filters use waveform amplitude, spectral power, and frequency-band energy to characterize audio.The appendix identifies x[n] as audio amplitude, P[k] as power spectral density, and E_f,t as frequency-band energy at a time frame.Source: §A.3 · p. 11
A.4 Definition of Evaluation Metrics
- The evaluation measures mean great-circle distance error and coarse-to-fine localization performance through thresholded accuracy at <1, <10, <100, and <1000 km.For a refusal, the distance is set to 10,000.Source: §A.4.1 · p. 11, §A.4.3 · p. 12
A.4.4 Reject Rate
- The appendix defines reject rate as the share of samples without a valid prediction and separately computes mean distance error for speech and non-speech samples.Source: §A.4.4 · p. 12, §A.4.5 · p. 12
A.5 Definition of Main Sound Labels
- The appendix defines Nature, Animal, Music, and Things as main sound labels, with Nature excluding animal and human sounds.Source: §A.5 · p. 12
A.6 Further Comparison Results
Further comparisons report model performance across sound categories and error percentiles, alongside experimental settings, error-type definitions, and the benchmark’s global distribution.
A.6 Further Comparison Results
- At the 25th percentile, Gemini 3 Pro’s localization errors are under 50 km across all four sound categories, while the best open-source model has a 634 km error at that percentile.Music and Animal categories have relatively lower errors; the authors suggest linguistic or distributional cues may contribute, while Nature and Sound of Things often lack such structure.Source: §A.6 · pp. 12–13
B.1 Global Distribution of AGL1K
- Figure 9 presents the global distribution of AGL1K, with markers for audio locations and colors indicating countries; its caption reports 1,444 audios across 74 countries.Source: Figure 9 · p. 13, §B.1 · p. 13
A.7 Further Experimental Settings
- Open-source models were run on one NVIDIA RTX 4090 GPU, with approximately 12 GPU hours of inference per model; closed-source models were accessed via APIs.Source: §A.7 · p. 13
B Definition of Each Error Type
- The error taxonomy includes bird bias, language ambiguity, over-commitment, over-reasoning, label misidentification, educated guess, and refusal.Language ambiguity includes correctly identifying a language spoken across multiple regions but selecting the wrong location; an educated guess is effectively a random response when information is insufficient.Source: B Definition of Each Error Type · p. 13
- Refusal errors are cases where a model declines to answer.Source: B Definition of Each Error Type · p. 13
B.2 Consistency on three contribution-labeling LLM
Contribution judgments from Gemini, Claude, and Qwen were consistent across individual samples and aggregated sound categories. The results support the stability of the ensemble-based annotation strategy, with some model-specific preferences.
B.2 Consistency on three contribution-labeling LLM
- Pairwise cosine similarity of contribution vectors ranged from 0.65 to 0.68 at the sample level, indicating substantial directional agreement on influential audio clues.The comparison measured agreement among Gemini, Claude, and Qwen.Source: §B.2 · p. 13, Figure 10 · p. 14
- Across model pairs, Pearson correlation for aggregated category contributions exceeded 0.91, while Top-10 category overlap remained above 0.60, showing agreement on most influential sound categories.These results indicate contribution judgments are stable across different LLMs, supporting the robustness of the ensemble-based annotation strategy while allowing minor model-specific preferences.Source: §B.2 · p. 14
- The authors characterize contribution judgments as stable across models while allowing for minor model-specific preferences.Source: §B.2 · p. 14
B.3 Continent-Distribution of Localizability
Across continents, average audio localizability differs and partially aligns with model performance; the appendix also describes human annotation and prompts used for contribution scoring and benchmark predictions.
B.3 Continent-Distribution of Localizability
- Africa, Asia, and South America had higher average localizability scores than Europe, North America, and Oceania; the authors suggest linguistic and cultural distinctiveness may contribute.Gemini 3 Pro's accuracy ranking across continents partially aligned with localizability, which the authors suggest may help predict regional geo-localization difficulty.Source: §B.3 · p. 14, Figure 11 · p. 14
B.4 Human Annotator
- During dataset refinement, a computer-science PhD student selected plausibly city-locatable examples; another annotator labeled failure cases, with linguistic advice for speech errors.Annotators were paid 100 RMB per hour and informed their annotations would be used in the study.Source: §B.4 · p. 14
Benchmark Prompt
- The benchmark prompt requires a location guess even for generic or uncertain audio and requests a JSON response with place names and coordinates.It also specifies English string values and latitude and longitude ranges.Source: Benchmark Prompt · p. 15
Contribution Calculation Prompts
- The contribution-calculation prompt asks models to assign 0.0–1.0 contribution scores to listed sound categories mentioned or implied in reasoning text.Scores map from not relevant at 0.0 to primary or dominant contribution at 1.0; only categories above 0.0 are included.Source: Contribution Calculation Prompts · p. 15
B.6 Sample Output
The appendix presents complete outputs for selected models on three representative audio cases and generates an image for each clip to aid interpretation.
B.6 Sample Output
- Complete outputs from selected models are presented for three representative audio clips.Source: §B.6 · p. 15
- Each audio clip is paired with an image generated using NanoBanana Pro to facilitate interpretation and understanding.Source: §B.6 · p. 15
B.6.1 Example 1-Gemini 3 Pro
In one example, Gemini 3 Pro combines environmental and linguistic clues to identify Essaouira, Morocco, while GPT-4o Audio uses the Adhan and urban noise to guess Istanbul, Turkey.
B.6.1 Example 1-Gemini 3 Pro
- Gemini 3 Pro identifies Essaouira, Morocco, by combining the Adhan, coastal wind, seagulls, French speech, and motorcycle noise.Its reasoning also notes heavy Atlantic surf and Arabic shouts.Source: §B.6.1 · p. 16
B.6.2 Example 1-GPT-4o Audio
- GPT-4o Audio guesses Istanbul, Turkey, from the Adhan and urban traffic, treating these cues as consistent with a Muslim-majority city.Source: §B.6.2 · p. 16
B.6.3 Example 1-Qwen3-Omni
The examples show models using combinations of transport, language, wildlife, and urban-environment cues to propose city locations. When cues are generic, the location prediction is explicitly speculative.
B.6.3 Example 1-Qwen3-Omni
- Qwen3-Omni combines a train or subway, ferry and waves, and Japanese speech to identify Kobe, Japan.Source: §B.6.3 · pp. 16–17
B.6.4 Example 1-Mimo-Audio
- Mimo-Audio treats train sounds and urban traffic as evidence for Mumbai, but calls the location an educated guess without more specific cues.Source: §B.6.4 · p. 17
B.6.5 Example 2-Gemini 3 Pro
- The Stockholm example uses a Swedish announcement naming the Uppsala route via Arlanda and Knivsta to locate the recording at Stockholm Central Station.The passage identifies the platform level as the recording location.Source: §B.6.5 · p. 17
B.6.6 Example 2-Qwen3-Omni
- Qwen3-Omni combines Korean, a KTX announcement mentioning Seoul Station, and train sounds to identify Seoul Station in South Korea.Source: §B.6.6 · p. 18
B.6.7 Example 3-Gemini 3 Pro
- Gemini 3 Pro combines courtyard reverberation, blackbird song, and nearby transit sounds to identify a residential neighborhood in Berlin.Source: §B.6.7 · p. 18
B.6.8 Example 3-GPT-4o Audio
- GPT-4o Audio describes generic bird, park, and distant-traffic sounds, then speculates that the location is New York City.Source: §B.6.8 · pp. 18–19
B.6.9 Example 3-Qwen3-Omni
- Qwen3-Omni uses song-thrush calls and urban traffic to propose London, United Kingdom.Source: §B.6.9 · p. 19