Source-linked AI summary
The WebKurator.de Platform: Combined Regional and Topical Web Curation
Michael Dinzinger, Natanael Arndt, Ben Böck, Jelena Mitrović, Michael Granitzer
TL;DR
National libraries need better ways to curate culturally and regionally relevant web content because directory-based systems intertwine topical and geographic information, while fully automated curation remains error-prone. WebKurator.de addresses this gap with a two-dimensional, collaborative platform that combines LLM-based classification, imprint-based address extraction, geocoding, and moderated user contributions. Its initial German foundation includes 2.58 million websites with topic labels and geographic information, while the architecture remains focused on the live German web and future archiving integration.
Problem
Existing web directories largely intertwine topical, regional, and linguistic information, while fully automated curation is insufficient for institutional-quality collections.
Method
WebKurator separates topic and region, automates topic classification and imprint-based address extraction with geocoding, and sends user suggestions through human moderation.
Results
2.58 million websites located in Germany have both topic labels and geographic information, forming the platform’s initial foundation.
Takeaways & Limitations
WebKurator combines scalable LLM-supported metadata generation with structured human moderation for regional exploration and selective web archiving.
Takeaways & Limitations
The current version focuses on the live German web, although its architecture is designed for future integration with selective web archiving workflows.
Abstract
from arXiv · showhide
The systematic curation of the Web remains a central challenge for national libraries and memory institutions that aim to preserve culturally and regionally relevant content. Existing directory-based approaches such as Curlie implement a predominantly topic-centric, one-dimensional hierarchy, where geographic aspects are intertwined with topical and linguistic categories. To address this limitation, we present WebKurator.de, a collaborative platform for combined regional and topical web curation, initially focused on the German web. WebKurator introduces a two-dimensional curation model that explicitly separates topical categorization and geographic annotation. The system integrates LLM-based topic classification and imprint-based address extraction with geocoding, and supports user suggestions together with moderated review. The platform is bootstrapped from the German Imprints Dataset, a large-scale collection of 5.54 million websites. Among them, 3.14 million contain imprint pages, for which we successfully extracted and geocoded postal addresses. Of these, 2.58 million (85.17%) are located in Germany and also have an assigned topic label. These websites form the initial foundation of WebKurator.de and can be continuously extended through user suggestions.
1 Introduction
WebKurator addresses the difficulty of curating culturally and regionally relevant websites when existing directories emphasize topical hierarchies and automated approaches remain error-prone. It proposes a regional and topical platform combining automated annotation with moderated user contributions.
- Motivation: National libraries need systematic web curation that covers both topical and regional dimensions.The German National Library’s collection mandate includes websites published in Germany, in German, or related to Germany.
- Limitations of Existing Approaches: Existing directories rely on labor-intensive, predominantly one-dimensional topical hierarchies that make combined regional and topical queries unintuitive.Curlie permits multiple categories and includes regional branches, but these remain embedded in a single hierarchy.
- Motivation: LLMs make automated website categorization and address extraction feasible, but fully automated curation remains insufficient for institutional-quality collections.The paper therefore combines automation with human oversight rather than treating automation as a complete replacement.
- Contribution: WebKurator introduces a two-dimensional model separating topic and region, with automated annotation and moderated user suggestions.Its workflow uses LLM-based topic classification and imprint-based address extraction followed by geocoding.
2 Related Work
Prior systems established the value of curated web collections but largely organized content through directory hierarchies or externally managed archiving workflows. WebKurator builds on these traditions while targeting their separation of topical and geographic information.
- Human-Curated Web Directories: ODP/DMOZ and Curlie provided community-maintained hierarchical directories for web discovery and research.ODP/DMOZ was discontinued in 2017, while Curlie continues the directory-based approach under an open license.
- Human-Curated Web Directories: Curlie models topic, region, and content language within one intertwined hierarchy rather than as independent facets.Topical branches, regional branches, and language-specific /World subtrees reflect historical design decisions retained for backward compatibility.
- Selective Web Archiving: Selective web archiving systems such as the Internet Archive and Archive-It provide curated access to preserved content through crawling, seed selection, and collection management.These systems focus on preserved web content rather than the live web.
- Selective Web Archiving: WCT and NetarchiveSuite support institutional archiving workflows, but generally assume seed selection and metadata creation occur externally or manually.Their workflows include selection, scheduling, harvesting, quality assurance, and WARC storage.
3 Collaborative Platform
WebKurator operationalizes separate topical and regional curation through automated metadata generation, geocoding, collaborative suggestions, and moderation. Its initial German Imprints Dataset provides large-scale coverage while retaining human review and exposing category-distribution differences from Curlie.
- 3 Collaborative Platform: WebKurator separates topical and regional dimensions, automates annotation, and uses moderation to quality-control newly suggested websites.The platform integrates address extraction, geocoding, topic classification, and moderated review.
- 3.1 Data Collection and Annotation: The initial dataset combines seed websites from the Open Web Index, Common Crawl, and Curlie with imprint-page detection and metadata collection.Landing pages and imprint pages are downloaded, with related pages retrieved where available.
- 3.1 Data Collection and Annotation: Qwen3-Next-80B extracts postal addresses and assigns topical categories, while Photon geocodes the extracted addresses locally.The topical labels use a curated subset of the FourSquare taxonomy.
- 3.2 Statistics: 5.54 million websites have valid landing pages, and 3.23 million have imprint pages.The .de country-code top-level domain accounts for 52.31% of the dataset.
- 3.2 Statistics: 3.14 million websites received geographic positions, with 85.57% located in Germany, 7.25% in Austria, and 5.19% in Switzerland.The positions derive from postal addresses extracted from imprint pages and geocoded with Photon.
- 3.2 Statistics: The German Imprints Dataset is more concentrated in Business and Professional Services than Curlie’s more balanced editorial-category distribution.The difference reflects an imprint-based strategy emphasizing legally registered organizations.
- Platform Functionality: The platform supports exploration, structured topic-and-region filtering, similar-website browsing, user suggestions, metadata edits, and moderation.Its current deployment focuses on the German subset but is designed for expansion to additional German-speaking countries.
4 Conclusion
WebKurator.de combines regional and topical web curation through separate geographic and topic information, enabling joint search. It integrates automated metadata generation with human moderation and currently focuses on the live German web while supporting future selective archiving.
- WebKurator.de enables combined regional and topical web curation by explicitly separating geographic and topic information.
- The platform integrates LLM-based topic classification, imprint-based address extraction, geocoding, and structured human moderation.
- 2.58 million websites in Germany have both topic labels and geographic information, forming the platform’s foundation for regional exploration and selective web archiving.
- WebKurator.de currently focuses on the live German web, while its architecture is designed for future integration with selective web archiving workflows.