PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Search

Fredrik Nygård Carlsen

arXiv:1509.02207v1cs.IRcs.DL

TL;DR

Growing information volumes and unsatisfactory CDS search ordering make relevant scientific publications harder to find. This work develops Obelix to re-rank results from existing search engines using user–item and user–user relationships, then evaluates it offline and online. For global CDS search, Obelix achieved a lower average click position than latest first and word similarity, while collection-specific preferences differed.

  • Problem

    CDS users were dissatisfied with search-result ordering, and generic ranking methods could require searching long result lists to find relevant items.

  • Method

    The work develops Obelix as a recommendation system that uses user relationships with items and other users to boost items in an existing search engine’s Original List.

  • Results

    Average click position for global search was 5.64 with Obelix, compared with 12.12 for latest first and 9.34 for word similarity.

  • Takeaways & Limitations

    Personalized search is recommended as the default for global CDS search, while each collection should be evaluated separately because preferred ranking methods differ.

  • Takeaways & Limitations

    d-Rank was not implemented in CDS because configuring it required implementation expertise that was lost after the author left CERN.

Abstract

from arXiv · show

As the volume of electronically available information grows, relevant items become harder to find. This work presents an approach to personalizing search results in scientific publication databases. This work focuses on re-ranking search results from existing search engines like Solr or ElasticSearch. This work also includes the development of Obelix, a new recommendation system used to re-rank search results. The project was proposed and performed at CERN, using the scientific publications available on the CERN Document Server (CDS). This work experiments with re-ranking using offline and online evaluation of users and documents in CDS. The experiments conclude that the personalized search result outperform both latest first and word similarity in terms of click position in the search result for global search in CDS.

1.1 Motivation

CDS users were dissatisfied with search-result ordering and often turned to Google, motivating personalized search based on users’ interactions. The project proposed and implemented Obelix to personalize CDS search and evaluate it with real users.

  • 1.1 Motivation: Long ranked result lists and simple ranking algorithms can force users to search several pages, affecting search experience and satisfaction.
  • 1.1 Motivation: The work targets digital libraries by learning user preferences automatically from search history and user–item relationships.
  • Data and scope: The study used CERN data, including ten years of logs and current CDS users and items; only logged-in user interactions were counted.
  • 1.4 Approach: The prototype was proposed, implemented, and tested with real CDS users after surveying information retrieval, personalization, and recommendation techniques.
  • 1.4 Approach: The project implemented Obelix, a recommendation system intended to personalize the CDS search experience.

Cornerstone modules of Invenio

Invenio combines modular digital-library components with information-retrieval methods for indexing, matching, scoring, and ranking items. Its search stack includes BibIndex, BibRank, WebSearch, and tf-idf-based vector similarity.

  • Cornerstone modules: Invenio is organized into relatively independent modules, including bibliographic, web-interface, search, import, classification, and access-control components.
  • Indexing: BibIndex uses forward and reverse indexes to associate words or phrases with records and records with indexed terms.
  • Ranking: BibRank creates record rankings from term frequency, freshness, popularity, and other parameters for use in search.
  • Search: WebSearch uses several indexes to provide fast responses and makes metadata directly browsable.
  • Vector Space Model: The vector space model represents items as n-dimensional vectors and scores them through cosine similarity between item and query vectors.
  • TF-IDF: Tf-idf weights terms by document frequency and corpus frequency, and search engines commonly use it to score document relevance.

2.2.3 h-index

The h-index summarizes a researcher’s citation distribution, while recommendation systems use user preferences and item content or trust relationships to generate personalized recommendations. These approaches support ranking and filtering but have stated scope constraints.

  • h-index: The h-index is the number h such that h papers have at least h citations each and the remaining papers have no more than h citations.
  • h-index: The h-index improves on citation-count-per-paper measures but may mislead when comparing scientists across different research communities.
  • Recommendation systems: Recommendation systems use user preferences to generate personalized item lists through collaborative or content-based filtering.
  • Content-based filtering: Content-based filtering compares item descriptors with a user profile built from items the user has seen.
  • Trust networks: Trust-network recommendations use trusted users’ recommendations, with propagation and aggregation estimating trust across users.
  • Trust networks: Trust networks can improve recommendation quality and help users form better-informed opinions through trusted users’ judgments.

Distrust

Distrust is distinct from low trust in recommendation networks and complicates trust propagation. The section also situates personalization and evaluation around user feedback, prediction accuracy, and ranking references.

  • Distrust: Most trust-network approaches ignore distrust or treat it as low trust, although distrust may not be equivalent to lack of trust.
  • Distrust: When distrust enters propagation chains, the same links can support trusting, distrusting, or ignoring information from another user.
  • Feedback: User relevance feedback can be explicit or implicit, with interactions such as clicks, viewing duration, scrolling, and browsing providing implicit signals.
  • Personalization: Personalization uses collected interaction data to tailor content and may improve search precision and user experience.
  • Evaluation: Recommendation evaluation commonly measures rating, usage, or item-ranking prediction accuracy.

Measuring Usage Prediction

Usage prediction evaluates whether a system can anticipate which items users will use, while ranking evaluation focuses on ordering search results by user preferences. The section describes reference-ranking and utility-based approaches, alongside assumptions and broader evaluation context.

  • Usage prediction: Usage prediction infers which items users may use from observed clicks, views, or downloads rather than explicit ratings.
  • Usage prediction: Offline usage evaluation hides some of a user’s selections and tests whether the system predicts those items.
  • Reference ranking: Usage data can rank used items above unused items only when users were aware of the unused items; observed versus used items can provide that distinction.
  • Ranking evaluation: Reference ranking compares a produced order with a correct or preference-based order, while utility-based ranking discounts item utility according to result position.
  • Ranking evaluation: Ranking evaluation may preserve ties because systems must return an ordered list even when the data does not reveal relative preferences.
  • Ranking evaluation: Utility-based ranking can model the likelihood that users observe an item at position i, assuming observation depends on position rather than item identity.

3.2.1 CDS Search Engine

CDS provides metadata-based search across diverse collections, while several ranking and retrieval extensions address scalability and relevance. Existing approaches include external search bridges, aggregated ranking, and citation-based graph ranking, each with scope or data limitations.

  • CDS search: CDS supports simple and advanced queries across all items or selected collections, returning results as one list or collection clusters.
  • CDS search: The current CDS search engine matches query words against metadata and disables full-text search because similar publications are difficult to distinguish.
  • Ranking approaches: The d-Rank project aggregates ranking signals including freshness, citations, tf-idf, and download history to improve search-result ordering.
  • Ranking approaches: Download frequency and freshness combined linearly outperformed every individual criterion tested, including download and view frequency, word similarity, and freshness.
  • Ranking approaches: d-Rank was not implemented in CDS because configuring it required implementation expertise that was lost when its author left CERN.
  • External search bridge: Solr was evaluated against Xapian and ElasticSearch, with Solr outperforming Xapian while ElasticSearch was considered too immature for production at that time.
  • Citation ranking: Citation-based graph ranking improves on citation counts, but its effectiveness is constrained by incomplete citation data in CDS.

3.3.1 User Satisfaction Survey

User studies identify search ranking, usability, and discoverability as major CDS concerns. Survey responses and interviews report frustration with search, reliance on Google, and difficulty finding older or popular items.

  • Survey scope: 150 users responded to the 2011 satisfaction survey, with two-thirds working at CERN and the remainder collaborating externally.
  • Search satisfaction: Only 32% said they could always find desired items with CDS search, while 52% disagreed.
  • Ranking preferences: 59% wanted result ordering by quality criteria such as downloads or citations, while 40% accepted the default latest-first ordering.
  • Interface use: 43% found the advanced interface difficult, 41% did not use it, and 15% found it easy; simple search averaged 19’053 daily uses versus 3’707 for advanced search.
  • Search experience: Users described CDS as a mandatory rather than discovery-oriented tool, and many considered Google faster or more reliable for reaching CDS content.
  • Librarian interviews: Librarians reported that popular older items could rank around 850 under latest-first ordering, requiring about 100 next-page clicks to find them.

3.3.4 Lessons learned

CDS faces challenges in ranking heterogeneous digital-library content: existing ordering methods do not consistently reflect users’ information needs. The proposed direction is to personalize ranking from user interactions and relationships rather than fixed groups or metadata alone.

  • Information retrieval must find relevant items in large collections and display them quickly in a satisfactory order.
  • Existing CDS ranking options include date, metadata, word similarity, and citation-based scoring.
  • Users and librarians reported dissatisfaction with CDS result ordering and preferred Google for finding items.
  • CDS contains heterogeneous collections, making word-similarity ranking less useful for distinguishing relevant items.
  • Interaction-based communities personalize recommendations by relating users who have used the same items, unlike communities defined by organizational groups.
  • Obelix complements keyword search by using user and item relationships to boost recommended items in the Original List.

4.2.1 List of requirements from the CDS team

The CDS team required Obelix to be configurable, self-learning, maintainable, resilient, and suitable for large-scale digital-library integration. The design therefore emphasizes generic interfaces, explainable interaction graphs, and efficient communication with the search system.

  • Requirements: Obelix must be easy to enable or disable, configure, install, evaluate, and maintain by IT staff.
  • Requirements: The system must preserve search speed, tolerate Obelix errors, and keep CDS functional if Obelix is unavailable.
  • Requirements: Obelix must support Scientific Linux, unique users and items, logs, open APIs, efficient communication, and multiple interaction types.
  • Requirements: Obelix must self-learn, start without information, and support importing millions of historical interactions to avoid cold start.
  • Architecture: Obelix is intended as a generic recommendation system for CDS and other search engines, not only a Python integration for Invenio.
  • Architecture: Decoupling Obelix from the IR system supports independent development and stable-API integration, but requires integration know-how and additional installation resources.
  • Data and evaluation: Recommendations use graph interactions, including item usage, click position, and search-result/item-usage relationships for operation and evaluation.
  • Recommendation model: The graph represents users and items; traversal from the current user uses distances and usage counts to calculate recommendation scores.

4.7.1 Building blocks

Obelix uses a modular architecture centered on a graph database and asynchronous processing. Events update the graph through queues, while recommendation lists are generated by traversal and stored for later retrieval.

  • Architecture: The architecture exposes queue and key/value-store interfaces between Obelix and the integrated IR system.
  • Graph database: The graph database stores user–item relationships and supports configurable Breadth-First traversal for recommendation generation.
  • Graph database: Traversal is the key graph-database feature because recommendation generation may require traversing a large graph.
  • Graph database: Neo4j outperformed all other evaluated candidates for traversal workloads regardless of workload or parameters.
  • Asynchronous processing: Obelix maintains separate insert and recommendation queues, with workers inserting events into the graph and triggering recommendation rebuilding.
  • Asynchronous processing: Queue-based handling increases event capacity because insertion is best effort and users need not wait for recommendation traversal.
  • Implementation boundary: The native queue is unsuitable for serious applications because it has limited performance, in-memory data loss risk, and no distributed scalability.

4.7.4 Key/Value Store

Obelix uses a key/value store to cache recommendations and support statistics, with Redis selected for the CDS implementation. Performance limitations of direct REST communication motivate a cached, queue-based setup.

  • Key/value store: Cached recommendations can be retrieved in constant time through a key/value store.
  • Key/value store: The native in-memory key/value store risks data loss on restart and cannot scale across several machines.
  • Key/value store: Obelix defines the key/value store as an interface, allowing deployments to choose among backends such as Redis, Voldemort, Dynamo, and Riak.
  • Key/value store: Redis was chosen for CDS because its binding was well documented and fast.
  • REST communication: The REST API supports all Obelix features, including recording interactions and requesting recommendations generated on demand or from cache.
  • REST communication: As recommendation volume grew, REST communication became a bottleneck for services such as CDS, despite partial relief from caching.
  • High-performance setup: A caching layer and queue allow asynchronous input handling, while recommendation insertion and fetching have time complexity O(1) under the stated setup.
  • Implementation: Java was selected for its libraries, ecosystem, deployment fit, and support for embedded Neo4j.

4.7.9 Logging

The evaluation logs user interactions and search results to assess Obelix’s efficiency and personalized re-ranking. Experiments combine offline simulation with online production data, while computational and evaluation constraints shape the offline design.

  • Logging: Obelix evaluation requires logging user clicks, search results, and the number of records viewed before finding a clicked item.CDS integration reports usage and search-result information to Obelix.
  • Re-ranking: The CDS integration scores original results from 0.0 to 1.0, joins them with recommendations, and re-ranks only items already in the original result.Obelix can combine recommendations with the existing latest-first ranking.
  • Evaluation design: Offline experiments use historical logs to simulate recommendations and predict clicks, whereas online experiments use live production interactions.The offline procedure assumes user behavior remains similar before and after introducing the new system.
  • Evaluation design: Offline evaluation is useful for filtering algorithms and parameters but cannot answer all system-performance questions, requiring online testing for the remaining issues.The authors identify offline evaluation’s narrow question set as a limitation.
  • Offline constraints: Computational limits prevent exhaustive simulation across 40 000 users, 1.4 million items, and about 250 million interactions, so interactions and time are sampled randomly.The sampling trade-off aims to minimize potential bias from using a subset of the data.

Results

The experiments examine baseline click positions, Obelix prediction parameters, and graph traversal. Latest-first ranking often requires substantial scrolling globally, while parameter tests identify shallow traversal as beneficial and show limited sensitivity to usage count.

  • Baseline results: Approximately 500 000 searches were collected over 21 days, and 10 000 randomly sampled searches were reproduced for evaluation.The daily sample size was 500 searches.
  • Baseline results: About 50% of users must scroll through more than 10 items under CDS’s default latest-first ranking to find the desired item.For these users, finding the desired item requires clicking the next-page link.
  • Baseline results: Searching within a collection makes desired items easier to find because collection selection filters irrelevant items and reduces the result set.The passage states that latest-first ranking is therefore more useful within a collection.
  • Baseline results: Photos and multimedia show higher variance than articles and books, consistent with less extensive metadata and weaker alignment between recency and user interest.A photo search may prioritize the best image rather than the newest one.
  • Parameter results: Usage counts from 25 to 200 produce almost identical prediction results, with similar performance for 25 and 100 usages and flattening around 150.The results do not identify a single best usage count, but favor focusing on recent usages.
  • Parameter results: Graph traversal depth 2 or 3 gives a clear gain, while performance flattens from depth 5 because most items are reached between depths 5 and 8.The authors relate excessive traversal to making all users part of the community.

5.3.1 Introduction

The online evaluation compares Obelix with latest-first and word-similarity rankings using users’ observed clicks. Its interpretation is constrained by changing collections, restricted items, anonymous permissions, and a re-search procedure that can bias results.

  • Evaluation setup: The evaluation compares Obelix with latest-first and word-similarity rankings by measuring how far users scroll before finding relevant items.Top-n evaluation considers the first ten items per collection by default on CDS.
  • Evaluation setup: The online experiment introduced recommendations as a new default ranking during the first two months of 2015, with settings varied dynamically for users.Users could disable recommendations by switching to latest-first or word similarity.
  • Evaluation setup: Searches were logged with users’ original views, then re-run using latest-first and word similarity to compare all three ranking methods.The comparison searches attempted to locate the item originally viewed by the user.
  • Limitations: Restricted items are omitted for anonymous users, requiring about 30% of searches to be skipped and reducing the test set.Different user permissions also make anonymous re-runs inaccurate for restricted content.
  • Limitations: Because CDS continuously adds items, re-run searches may produce different results from the original searches.This creates inaccuracy when evaluating historical interactions against later result sets.
  • Limitations: Re-running searches can bias evaluation toward recommendations because the scroll-based measure assumes users would click the same relevant item under every ranking.Different result orders might lead users to select different items.

Result

The project developed and evaluated Obelix, a personalized search system that re-ranks existing CDS results without adding or removing items. Online experiments found improved global-search ranking, while collection-specific performance varied.

  • Online results: In global search, rec[0,1] outperformed latest first and word similarity, and click position improved during the online experiment.The comparison is shown in the ranking-methods evaluation.
  • Evaluation: Offline experiments predicted about 45% of user interactions using one year of collected data, motivating a fully functional implementation.The offline experiments were used to identify a promising combination of parameters before deployment.
  • Online results: Average click position was 5.64 with Obelix, compared with 12.12 for latest first and 9.34 for word similarity in global search.For individual collections, CERN Yellow Reports preferred latest first, while books performed best with word similarity.
  • Limitations and future work: The personalized search engine could not replace latest first and word similarity in all use cases, so collections require separate evaluation.The project recommends keeping personalized search enabled by default for global search and improving collection-specific ranking in future work.
  • Implementation: Obelix was developed from scratch, including its ranking algorithm, REST API, data import, and swappable backend technologies.The system was integrated with Invenio as an optional recommendation module and included caching and statistics infrastructure.
  • Method: Obelix re-ranks existing search results rather than introducing or removing items from the original result set.This design supports integration with an existing service such as CDS.

AppendixA

Appendix A describes Invenio’s bibliographic, indexing, search, administrative, and user-facing modules. Together, these modules support metadata processing, retrieval, access control, alerts, storage, and collaboration.

  • Bibliographic modules: BibCheck, BibConvert, BibEdit, BibFormat, BibUpload, and BibQuality support metadata validation, conversion, editing, formatting, and loading.BibConvert handles configurable structured or semi-structured format conversion, while BibFormat prepares bibliographic records for outputs such as search results.
  • Indexing and retrieval: BibIndex maintains word and phrase indexes, including forward and reverse mappings between indexed terms and record identifiers.Logical indexes can be configured for fields such as authors and titles.
  • Indexing and retrieval: BibRank configures ranking criteria for the search engine, including word frequency, metadata values, journal impact factor, and download counts.The module operates independently of BibIndex.
  • Administration: WebAccess uses role-based access control, while BibSched coordinates bibliographic database tasks and administrative queue operations.BibSched helps control database access and coherent execution of update tasks.
  • User-facing modules: WebAlert, WebBasket, WebComment, WebMessage, and WebHelp provide alerts, personal or shared document storage, commenting, communication, and documentation.WebAlert can send notifications by email or store results in baskets, while WebBasket supports private or public collections.
  • Indexing and retrieval: WebSearch supports word, phrase, Boolean, regular-expression, metadata, reference, and full-text searches, and can propose alternative matches.Users can also browse available index terms.
Loading 1509.02207v1…