PaperlayerView exact revision on arXiv

Source-linked AI summary

The Serendiptionnaire: a Questionnaire for Measuring Experienced Serendipity

Brett Binst, Annelien Smets

arXiv:2610.29487v1cs.HC

TL;DR

Experienced serendipity is hard to measure, and a validated questionnaire for it was missing. The authors developed the Serendiptionnaire through cognitive interviews and iterative item revision. Respondents of varied backgrounds understood the final items as intended, though the instrument’s scope and completeness remain bounded.

  • Problem

    A validated measure of experienced serendipity was missing, while existing operationalizations make comparison and synthesis across studies difficult.

    Source: §2.3 · p. 5, §6 · p. 21

  • Method

    The authors used twenty-seven cognitive interviews across three iterations to develop and refine questionnaire items until respondents interpreted them as intended.

    Source: §1 · p. 2, §4.1 · p. 10, §7 · p. 24

  • Results

    Respondents with varied ages, education levels, and native languages understood the Serendiptionnaire’s items as intended.

    Source: §7 · p. 24

  • Takeaways & Limitations

    The Serendiptionnaire fills the identified gap in a validated questionnaire for measuring experienced serendipity in RecSys research.

    Source: §6 · p. 21

    The authors do not consider the instrument exhaustive and identify additional conditions that could be added, including synchronicity.

    Source: §6.3 · p. 24

Abstract

from arXiv · show

Serendipity is important to design for, yet it remains difficult to measure and evaluate. No validated questionnaire exists, so studies either borrow ResQue's novelty scale, which does not measure serendipity, or build their own items, which makes findings hard to compare and synthesize. Building on our earlier conceptualization of experienced serendipity as a fortuitous encounter with content that is both refreshing and enriching, we developed the Serendiptionnaire, a questionnaire that operationalizes our framework. We conducted twenty-seven cognitive interviews over three iterations with respondents who varied in age, education level, country of birth, and English proficiency, while they browsed recommendations on platforms such as Instagram, TikTok, YouTube, and Spotify. After each iteration, we revised the items until respondents interpreted them as intended. The resulting instrument consists of modular scales that researchers can combine to target the specific flavor of serendipity they want to study or design for. The Serendiptionnaire offers a ground truth for validating system-centered serendipity metrics and LLM-based evaluations, and it provides a nuanced evaluation of users' experienced serendipity.

1 Introduction

Experienced serendipity is difficult to measure because it is subjective and can take different forms, while existing measures do not reliably capture those differences. The paper develops the Serendiptionnaire through cognitive interviews to make the concept measurable with items respondents understand as intended.

1 Introduction

  • Experienced serendipity is a fortuitous encounter with content that is both refreshing and enriching, with different conditions producing distinct flavors.Examples include an encounter that resolves a latent need and one that creates a new interest.Source: §1 · pp. 1–2
  • A validated questionnaire for experienced serendipity was missing, despite being considered essential for research progress.The authors argue that evaluation should listen to users directly through a standardized instrument [27] [50].Source: §1 · p. 2
  • ResQue is often used as a proxy, but its novelty scale cannot distinguish serendipity from mere unfamiliarity or separate its different flavors.ResQue does not include a designated serendipity item.Source: §1 · p. 2
  • The authors used cognitive interviews to revise questionnaire items when respondents’ interpretations diverged from the intended meaning.Cognitive interviewing exposes how respondents process items, a process that can drive answers and response error.Source: §1 · p. 2
  • Twenty-seven cognitive interviews produced the Serendiptionnaire, intended to support cross-study comparison, validate system-centered metrics, and evaluate users’ experienced serendipity.The authors also identify potential uses in LLM evaluation and stimulation of experienced serendipity in RecSys.Source: §1 · p. 2

2 Background

The background distinguishes design intent, system affordances, and users’ actual experiences, then frames experienced serendipity as a composite concept with multiple expressions. It reviews prior measures and explains why a validated instrument for experienced serendipity is still needed.

2.1 Intended, Afforded, and Experienced Serendipity

  • The framework separates intended serendipity, the design goal; afforded serendipity, system capacities; and experienced serendipity, the user’s actual experience.These levels are intertwined, but afforded capabilities yield experienced serendipity only when users can actualize them.Source: §2.1 · pp. 2–3

2.2 Experienced Serendipity

  • Experienced serendipity is a fortuitous encounter with content that is both refreshing and enriching, and it can take different forms.The framework treats each component as fulfillable through different conditions, producing distinct flavors.Source: §2.2 · p. 3, Figure 2 · p. 4

2.2.1 Fortuitous.

  • Fortuitous encounters can be unintentional, unexpected, difficult to find, or uncovering something the user had not previously seen.Difficult-to-find includes content perceived as a hidden gem or as harder to encounter without the platform.Source: §2.2.1 · p. 4

2.2.2 Refreshing.

  • Refreshing conditions include novelty, unusualness, and reconnecting with the past; novelty and unusualness are distinct, since familiar content can still differ from usual tastes.Taste-reincarnation covers rediscovery or new content that recalls an earlier period.Source: §2.2.2 · p. 4
  • The enriching component concerns a positive effect, including curiosity or interest, positive emotional response, or meaning aligned with the user’s values.Intriguing content may open a new interest, while resonance can involve positive emotion or perceived alignment.Source: §2.2.2 · p. 5
  • Other refreshing conditions include energizing, inspiring, impactful, empowering, useful, or resonant encounters.These range from motivation or restored energy to usefulness for a goal, affirmation, or a positive emotional response [42].Source: §2.2.2 · p. 5

2.3 Measuring Serendipity

  • ResQue’s novelty scale is used to measure experienced serendipity, but novelty alone cannot distinguish unfamiliar content from serendipity or capture its different flavors.Other studies use varied operationalizations, making comparison and synthesis difficult.Source: §2.3 · pp. 5–6
  • Prior questionnaires converge on experienced serendipity as a composite concept with varied criteria and expressions, rather than one uniform experience.The authors use this framework to operationalize different notions of experienced serendipity in a questionnaire.Source: §2.3 · pp. 6–7

3 Methodology

The methodology uses reparative cognitive interviews to align respondents’ interpretations with the intended framework. The researchers sampled across user characteristics and platforms, iteratively tested the questionnaire, and analyzed interview feedback to revise it.

3.1 Cognitive Interviewing Approach

  • Cognitive interviewing was chosen to maximize the likelihood that respondents interpret questionnaire items as intended.The authors adopt a reparative approach, which moves respondents’ interpretations toward the researcher’s intended meaning.Source: §3.1 · p. 7, §3.3 · p. 8

3.2 Sampling Approach

  • The study conducted three interview iterations with nine respondents each, purposively sampling across age, education, country of birth, and English proficiency.The questionnaire was developed across multiple platforms, with participant and platform details listed in Table 2.Source: §3.2 · p. 7, Table 2 · p. 7

3.3 Interview Procedure

  • Respondents browsed a familiar recommender platform before completing the questionnaire while thinking aloud, with probing used when further insight was needed.Probes explored reasoning, interpretations of key terms, hypothetical situations, and item clarity across respondents with different English proficiency and education levels.Source: §3.3 · p. 8

3.4 Analysis

  • Researchers transcribed and coded interviews using deductive appraisal codes and inductive codes for issues and interpretations outside that framework.Each iteration coded transcripts, summarized item feedback by scale, and used the findings to adapt the questionnaire.Source: §3.4 · pp. 8–9, Table 3 · p. 9

4 The Serendiptionnaire

This section documents the refinement of the Serendiptionnaire’s items and instructions across three iterations, including how respondents interpreted misleading and resonance items and how the resonance scale was revised. It preserves distinctions among encounter types and between emotional reaction and personal meaning, while noting where researchers must specify a referent or study context.

4.1 Introduction to the Serendiptionnaire

  • Respondents who read the revised introduction understood its focus on feelings about content from the session, but many paid it little attention.The introduction was added in iteration two and shortened in iteration three; it emphasizes the session just completed and the respondent’s feelings about its content.Source: §4.1 · p. 10

UNINTENDED; DIFFICULT TO FIND

  • More specific unintended items performed better, while “hidden gem” and difficulty discovering content without a platform measured distinct judgments.Users could confuse the session’s goal with whether an individual encounter was intended, so the referent should be specified. Explaining “hidden gem” clarified it for respondents with weaker English, but some content was a hidden gem yet easy to find elsewhere, or difficult to find without being a hidden gem.Source: UNINTENDED · p. 10, DIFFICULT TO FIND · p. 11

UNEXPECTED

  • Respondents generally understood surprise and unexpectedness as intended, but sometimes distinguished them and compared encounters against different standards.Predictability was dropped as a synonym of expected; “expected to find” remained ambiguous and should be adapted to context. Respondents compared encounters either with prior platform interactions or with the content itself.Source: UNEXPECTED · pp. 11–12

UNCOVERING; NOVELTY

  • Uncovering items were generally understood as intended, while novelty items were concrete and easy to answer; both scales benefit from specifying the item or analysis level.Uncovering spans awareness, usefulness, and noticing details, which respondents perceived as distinct or hierarchical; some abstract wording remained difficult. Novelty wording distinguishes not having seen an item from not knowing about it, rather than measuring unusualness.Source: UNCOVERING · p. 12, NOVELTY · p. 13

UNUSUAL; TASTE-REINCARNATION

  • Unusualness items were generally interpreted as intended, while taste-reincarnation items retained ambiguity about what “the past” meant.The unusualness scale dropped horizon-broadening wording because it implied enrichment, which is measured separately. Nostalgia was reframed as subjective feeling, and reconnection as returning to something previously liked.Source: UNUSUAL · p. 14, TASTE-REINCARNATION · p. 14

INTRIGUE; ENERGY

  • Interest and curiosity items were understood as intended despite differing interpretations, while energy items distinguished restoration from motivation.The motivation item was made a “Motivated me to ...” template because motivation needs a direction; “couch potato” received an explanatory definition. Respondents read “recharged” as a short restorative break and “gave me energy” as more general than motivation.Source: INTRIGUE · p. 15, ENERGY · pp. 15–16

INSPIRATION; IMPACT

  • Inspiration was interpreted more broadly than creativity, and respondents said its items should specify the study’s goal; impact items conveyed graded positive effects and distinct negative meanings.Creativity was associated with art and hobbies, whereas inspiration extended to areas such as study, career, and social life. “Toxic” and “harmful” concern negative impact rather than negative feeling, and wasting time can indicate opportunity cost.Source: INSPIRATION · p. 16, IMPACT · p. 17

EMPOWERING; USEFUL

  • The final empowering scale uses self-esteem items respondents understood as intended, while usefulness items focus narrowly on instrumental help.Self-esteem replaced earlier empowering, capability, and confidence formulations that were unclear, vague, too strong, or inapplicable. Wasted-time wording refers to time spent on a specific task in the usefulness scale, making it most informative in task-driven settings but still usable for background serendipity during aimless browsing.Source: EMPOWERING · p. 18, USEFUL · p. 18

USEFUL

  • Respondents understood misleading content as content warranting suspicion or distrust, matching the intended meaning.One participant connected misleading content with skin-care advertisements making claims about lotion.Source: USEFUL · p. 18

RESONANCE

  • The resonance scale moved “toxic” to impact and dropped “superficial” because respondents often did not understand it.“Empty” and “superficial” were both tested to express lack of resonance; “toxic” was read as negative change rather than negative affect.Source: RESONANCE · p. 19
  • Resonance captures emotional response through “touching” and positive alignment with personal values through “meaningful”; respondents indicated these facets need not co-occur.A participant found kittens touching and women’s-rights videos meaningful, while another rated an Albanian-history podcast meaningful but not touching or moving.Source: RESONANCE · p. 19

5 Manual of the Serendiptionnaire

The manual explains how to select and score scales for a target serendipity flavor, adapt items to the study context, and choose a shared referent for judging its characteristics.

5.1 Modular: Choosing the Scales that Characterize the Flavor(s) of Interest

  • Researchers select only the scales defining their target serendipity flavor, which should be clearly specified.For taste-broadening serendipity, the example combines unusual, intriguing, and fortuitous scales.Source: §5.1 · p. 19

5.2 Scoring Experienced Serendipity

  • A scale is satisfied when any item reaches “somewhat agree”; a flavor requires all its defining characteristics to hold together.The example classifies taste-broadening serendipity when at least one fortuitous condition, one unusual item, and one intriguing item are fulfilled; its final score is binary.Source: §5.2 · pp. 19–20

5.3 Adapting the Items to the Research Context

  • Adapt abstract items to the intended experience and how the design affords it, specifying the relevant goal or activity.Examples adapt “inspired me” to cooking something healthy and “motivated me” to exercising.Source: §5.3 · p. 20
  • Fix one referent for all characteristics, since they must be satisfied by the same object for a flavor to count.Otherwise, respondents could judge fortuitousness and enrichment against different objects.Source: §5.3 · p. 20

5.3.1 The Recommendation as Referent.

  • For recommendation-level evaluation, assess each scale against the same recommendation; a flowchart may reduce repetition by screening with core characteristics first.For emotionally resonant book recommendations, resonance can be screened first, with other scales skipped when a book scores negatively.Source: §5.3.1 · pp. 20–21

5.3.2 The Session as Referent.

  • A session can be the referent when serendipity lies in a recommendation sequence, but pilot whether respondents can evaluate the session as a whole.A song path toward a target genre is an example where taste-broadening serendipity may be best measured at session level.Source: §5.3.2 · p. 21

6 Discussion

The discussion frames the Serendiptionnaire as a validated response to fragmented measures and describes its uses for evaluating metrics, refining designs, and updating the conceptual framework.

6 Discussion

  • The Serendiptionnaire addresses fragmented operationalizations that impede comparison, synthesis, and progress in RecSys serendipity research.The authors present it as filling the gap left by the absence of a validated questionnaire.Source: §6 · p. 21

6.1 Empirical Implications: a Ground Truth for Validating Offline Metrics of Serendipity

  • Researchers can validate metrics against specific Serendiptionnaire scales rather than an undifferentiated serendipity judgment.The proposed approach targets metrics to particular flavors or dimensions of experienced serendipity.Source: §6.1 · pp. 21–22
  • The Serendiptionnaire standardizes vocabulary so researchers can compare findings and synthesize work across studies.It also makes metric development a targeted empirical problem by testing whether metrics predict relevant scales.Source: §6.1 · p. 22
  • The framework and items can serve as a benchmark and prompt context for LLM evaluation of particular serendipity flavors.The authors suggest this use might improve LLM performance in evaluating and stimulating experienced serendipity.Source: §6.1 · p. 22
  • When unusual recommendations fail to trigger intrigue, the questionnaire can indicate that designers should target intrigue rather than novelty alone.The instrument is presented as providing granular feedback on why an intervention did not increase serendipity.Source: §6.1 · p. 22

6.2 Theoretical Contribution: Updates to Our Conceptualization of Experienced Serendipity

  • The updated conceptualization makes unintentionality optional, broadens uncovering to discovery from a new perspective, and permits negative scores on enriching.Uncovering is a cross-cutting condition of fortuitous and refreshing; the framework no longer rules out mixed positive and negative experience.Source: §6.2 · pp. 22–23, Table 4 · p. 23

6.3 Limitations and Future Studies

  • The authors do not consider the framework exhaustive, noting that synchronicity and further enriching conditions may be added.Source: §6.3 · p. 24
  • The questionnaire was developed for evaluating experiences with RecSys; its applicability to other digital systems remains an open question.Source: §6.3 · p. 24

7 Conclusion

The conclusion presents the Serendiptionnaire as an interview-refined questionnaire for measuring experienced serendipity. It highlights uses in metric validation, LLM evaluation, and diagnosing design outcomes.

7 Conclusion

  • Twenty-seven cognitive interviews across three iterations refined the questionnaire until respondents with varied backgrounds understood its items as intended.The instrument operationalizes experienced serendipity as conceptualized in the authors’ earlier work.Source: §7 · p. 24
  • The Serendiptionnaire lets researchers test system-centered metrics as predictors of relevant scales and compare LLM evaluations.Source: §7 · p. 24
  • Its fine-grained feedback can help explain why a design succeeds or fails to stimulate experienced serendipity.Source: §7 · p. 24

8 Ethical and Human Subjects Considerations

The study protocol received ethics committee approval, participants provided informed consent, and they were told they could withdraw at any time without penalty.

8 Ethical and Human Subjects Considerations

  • The institute’s Ethics Committee reviewed and approved the study protocol.Source: §8 · p. 24
  • All participants gave informed consent before the interviews.Source: §8 · p. 24
  • Participants were told they could withdraw at any time without penalty.Source: §8 · p. 24

A The Full Serendiptionnaire

The appendix identifies its content as the final version of the Serendiptionnaire described in Section 4.

A The Full Serendiptionnaire

Introduction

The questionnaire asks respondents to report how they feel about content encountered during the session they just finished, answering from their first impressions.

Introduction

  • Respondents are told there are no right or wrong answers and are asked not to overthink their responses.Source: Introduction · p. 28

UNINTENDED

This section presents questionnaire items for assessing unintended discoveries and their personal, emotional, and practical effects. It also includes paired response items measuring empowering effects, perceived usefulness, and resonance during a platform session or content encounter.

USEFUL; RESONANCE

  • Usefulness items cover help with current or future problems, while resonance items capture content that feels meaningful, moving, or touching.The scales also include potentially misleading content and content experienced as annoying, disturbing, or empty.Source: USEFUL · p. 32, RESONANCE · p. 32

B Iteration 1

  • The first tested iteration’s questionnaire had no introduction.The paper identifies it as the first of three iterations through which the Serendiptionnaire was tested and refined.Source: B Iteration 1 · p. 33

UNINTENDED; DIFFICULT TO FIND; UNEXPECTED

  • Items assess whether encounters were unplanned, difficult to find independently, or surprising.They include encountering content not on purpose, a hidden gem, platform-assisted discoveries that would be difficult to find alone, and content that took users by surprise.Source: UNINTENDED · p. 33, DIFFICULT TO FIND · p. 33, UNEXPECTED · p. 33

UNCOVERING; NOVELTY; UNUSUAL

  • Scales distinguish self-discovery from unfamiliar content and content that differs from usual tastes.Uncovering includes new self-understanding and realizations, and is noted to cross-cut the fortuitous and refreshing component; novelty covers unfamiliar content, while unusualness covers content outside typical choices or habits.Source: UNCOVERING · pp. 29, 33, 34, NOVELTY · p. 34, UNUSUAL · p. 35

TASTE-REINCARNATION; INTRIGUE

  • Taste-reincarnation items cover nostalgia and reconnection with past interests, while intrigue items capture curiosity and fascination.The item wording also allows for reduced interest in things once enjoyed and encounters that leave users uninterested.Source: TASTE-REINCARNATION · p. 35, INTRIGUE · p. 35

ENERGY; INSPIRATION

  • Energy and inspiration items measure whether a session energized or motivated users and sparked ideas, imagination, or creativity.The item formulations also include recharging users and inspiring them.Source: ENERGY · p. 35, INSPIRATION · p. 36

IMPACT; EMPOWERING

  • Impact items assess possible positive effects on life and personal growth, while empowering items cover confidence, capability, control, and self-belief.The formulations contrast these with negative life impacts and disempowerment.Source: IMPACT · p. 36, EMPOWERING · p. 36

EMPOWERING

  • The EMPOWERING items contrast disempowerment, diminished confidence, incapability, reduced control, and self-doubt with their empowering opposites.Respondents rate the contrast from “Disempowered me” to “Empowered me.”Source: EMPOWERING · p. 36

USEFUL

  • USEFUL items assess whether encountered content and the platform session felt harmful, toxic, or misleading versus useful, helpful, or valuable.The passage provides separate item sets about encountered content and the session overall.Source: USEFUL · p. 37

RESONANCE

  • RESONANCE items contrast out-of-touch, alienating, empty, or indifferent experiences with being touched, resonating, finding meaning, or being moved.The subsection includes separate contrasts for the session overall and for encountered content.Source: RESONANCE · p. 37

C Iteration 2

The provided questionnaire versions ask respondents to rate how platform content felt, using items about discovery, novelty, usefulness, and emotional or motivational effects. The versions label the second and third refinement iterations and invite gut or first-impression responses.

C Iteration 2; Introduction; D Iteration 3

UNINTENDED; DIFFICULT TO FIND; UNEXPECTED

UNCOVERING; NOVELTY

  • Discovery items ask whether content revealed something new or useful, surfaced previously unnoticed details, or included items respondents had not encountered before.Source: UNCOVERING · pp. 39, 45, NOVELTY · p. 39

UNUSUAL; TASTE-REINCARNATION

INTRIGUE; ENERGY

INSPIRATION; IMPACT

USEFUL; EMPOWERING

RESONANCE

  • Resonance items contrast content that feels meaningful, moving, or touching with content that feels toxic, disturbing, empty, or superficial.Source: RESONANCE · pp. 43, 48
Loading 2610.29487v1…