PaperlayerView exact revision on arXiv

Source-linked AI summary

Sniff AI: Is My 'Spicy' Your 'Spicy'? Exploring LLM's Perceptual Alignment with Human Smell Experiences

Shu Zhong, Zetao Zhou, Christopher Dawes, Giada Brianz, Marianna Obrist

arXiv:2411.06950v1cs.CLcs.HC

TL;DR

Human-AI perceptual alignment in olfaction remains underexplored, despite its relevance to how AI interprets human sensory experience. The paper tests scent understanding with an LLM-based system and 40 participants; results show limited alignment, with scent-specific biases and modest success rates.

  • Problem

    Human-AI alignment of olfactory judgments remains unexplored, leaving open whether AI can interpret scent descriptions in ways that match human sensory experience.

    Source: §1 · pp. 1–2

  • Method

    A 40-participant study used an LLM-based embedding system to guess scents from descriptions and assess scent relationships through interactive tasks.

    Source: §1 · p. 2, §6 · p. 16

  • Results

    Success was 27.50% in Task 1 and 37.50% in Task 2, with scent semantics represented only to a limited extent and alignment biased toward certain scents.

    Source: §1 · p. 2, §6 · p. 16

  • Takeaways & Limitations

    The findings indicate that LLM embeddings capture some scent-related semantics, but accuracy is limited and predictions favor certain scents.

    Source: §1 · p. 2, §8 · p. 19

    The scent sample was limited and based on the Fragrance Wheel, so broader scent diversity remains outside this study’s scope.

    Source: §7 · p. 18

Abstract

from arXiv · show

Aligning AI with human intent is important, yet perceptual alignment-how AI interprets what we see, hear, or smell-remains underexplored. This work focuses on olfaction, human smell experiences. We conducted a user study with 40 participants to investigate how well AI can interpret human descriptions of scents. Participants performed "sniff and describe" interactive tasks, with our designed AI system attempting to guess what scent the participants were experiencing based on their descriptions. These tasks evaluated the Large Language Model's (LLMs) contextual understanding and representation of scent relationships within its internal states - high-dimensional embedding space. Both quantitative and qualitative methods were used to evaluate the AI system's performance. Results indicated limited perceptual alignment, with biases towards certain scents, like lemon and peppermint, and continued failing to identify others, like rosemary. We discuss these findings in light of human-AI alignment advancements, highlighting the limitations and opportunities for enhancing HCI systems with multisensory experience integration.

1 Introduction

The paper frames olfactory perceptual alignment as an unexplored part of human-AI alignment and asks whether LLMs can interpret human scent descriptions. It investigates this question through an interactive scent-guessing system and a 40-participant study, finding limited alignment biased toward certain scents.

1 Introduction

  • Olfactory alignment remains unexplored despite perceptual alignment’s importance for reliable, beneficial AI applications in everyday settings.The paper asks whether AI can interpret scent descriptions in ways that resonate with personal sensory experiences.Source: §1 · pp. 1–2
  • LLM embedding spaces encode semantic similarities and differences, motivating a test of whether their scent relationships align with human perceptions.The system maps human language descriptors into the embedding space and suggests scents based on semantic similarity.Source: §1 · p. 2
  • A 40-participant interactive study found scent semantics represented to a limited extent, with alignment biased toward certain scents and characteristics.Examples include confusing a description of rosemary with eucalyptus and correctly identifying an “intense masculine scent” as oakmoss.Source: §1 · p. 2

2 Background and Related Work

Prior work studies human-AI and representational alignment, olfaction’s role in HCI, and computational approaches to smell. The paper positions LLM sentence embeddings as a way to examine context-rich scent descriptions, while noting that olfactory alignment remains underexplored.

2.1 Human-AI Interaction and Perceptual Alignment

  • Studies of visual, tactile, and other sensory judgments provide precedents for alignment research, but human-AI olfactory alignment remains unexplored.Prior work reported varying findings, including detached MLLM descriptions of tactile qualities and GPT-4 judgments correlated with human judgments for some sensory inputs.Source: §2.1 · pp. 2–3
  • Embedding-space analysis can examine how models organize scent similarities and relationships, extending earlier work on representational alignment.The paper focuses on LLM representations of human smell experiences and their scent relationships.Source: §2.1 · p. 3

2.2 Growing Importance of Smell in HCI and Experience Design

  • Smell can enrich HCI experiences through immersion, attention shifts, memory, spatial orientation, and wellbeing, and HCI toolkits support olfactory prototyping.Smell & Paste enables novices to create personalized scratch-and-sniff experiences; OWidgets supports creating and replicating olfactory experiences.Source: §2.2 · p. 3

2.3 Bridging Human and Machine Understanding of Smell Experiences

  • Scent lacks a standardized vocabulary, and prior olfaction work includes molecular prediction, smell-language datasets, and chemical sensing.The paper notes that everyday users may find chemical names difficult to use for identifying scents.Source: §2.3 · pp. 3–4
  • Sniff AI processes a human scent description through ASR and embedding comparison, then predicts and delivers a scent.The comparison is with pre-built scent embeddings; the paper also describes an AI guessing mechanism and scent delivery device.Source: Figure 2 · p. 4
  • LLM sentence-level embeddings represent whole, context-dependent descriptions rather than static individual words, supporting analysis of scent meaning and sentiment.The paper presents this approach as richer and more context-aware than word embeddings for nuanced, multidimensional scent descriptions.Source: §2.3 · p. 4

3 The Sniff AI System Design

This section introduces Sniff AI as a system for testing whether LLMs understand human-described smells through vector relationships. Its five components take a description through speech recognition, embedding-based processing and scent guessing, then deliver a scent.

3 The Sniff AI System Design

  • Sniff AI investigates whether LLMs understand human scent descriptions by analyzing vector relationships in latent embedding space.The section introduces the system design and how LLMs encode scent descriptions as high-dimensional vectors.Source: §3 · p. 4
  • The system’s five components include ASR, pre-built scent embedding generation, mapping human smell experiences into AI embedding space, scent guessing, and scent delivery.The section also describes how the scents were selected.Source: §3 · p. 4
  • The component workflow is designed to process a human description and produce a guessed scent for delivery.The section directs readers to subsequent sections for the implementation details of each component.Source: §3 · p. 4

3.1 Scent Embedding Space and the Sniff AI System Overview

The paper treats LLM embeddings as a way to inspect how scent descriptions are organized and related. It proposes that spatial relationships among scent vectors should reflect human-perceived similarities and differences, then evaluates those ideas in two scent-description tasks.

3.1 Scent Embedding Space and the Sniff AI System Overview

  • Encoder-based embedding models are presented as a way to study how LLMs organize concepts and relationships, including scent descriptions.The paper contrasts these models’ interpretive role with decoder-only models’ autoregressive output generation.Source: §3.1 · p. 4

3.1.1 Scent Embedding Space

  • If AI perception aligns with human experience, similar scents should be near one another in embedding space, while distance and direction represent differences.The paper illustrates this with lemon and lime as similar citrus scents and rose and jasmine as similar floral scents.Source: §3.1.1 · pp. 4–5

3.1.2 An overview of the "Sniff AI" study and system

  • The study uses Sniff AI to evaluate alignment through a Scent Description Task and an Interactive Scent Comparison Task.The system recognizes voice input, predicts scents, and delivers them in real time.Source: §3.1.2 · p. 5

3.2 Pre-encoded Scent Embeddings

The study converts scent catalogue descriptions into vectors in the LLM’s embedding space, creating representations used as the system’s scent knowledge.

3.2 Pre-encoded Scent Embeddings

  • The descriptions are converted into numerical vectors in the LLM’s high-dimensional embedding space as pre-encoded scent embeddings.Source: §3.2 · p. 5
  • Catalogue descriptions encode each scent’s source and composition, including the plant source and extraction method for rosemary essential oil.Domain experts helped create the descriptions.Source: §3.2 · p. 5
  • The process produces 20 vectors, one for each scent, which form the basis of the AI’s scent knowledge and are reused throughout the study.The vectors are referenced in Table 1; scent concentrations, manufacturers, and catalogue descriptions are in the supplementary materials.Source: §3.2 · p. 5

3.3 ASR and Mapping Human Olfactory Experiences to the AI Embedding Space

The study maps participants’ spoken descriptions of scents into the LLM’s embedding space to examine scent matching and scent-to-scent transitions. The two tasks differ in whether participants describe one scent or compare two.

3.3 ASR and Mapping Human Olfactory Experiences to the AI Embedding Space

  • Both tasks encode participants’ descriptions of smell experiences to represent the LLM’s olfactory perception in its embedding space.The descriptions are captured and transcribed with OpenAI’s whisper-1 ASR, then encoded with the same text-embedding-3-large encoder used for scent representations.Source: §3.3 · p. 5

3.3.1 Task 1: Scent Description

  • Task 1 tests whether the encoder can match a specific scent from a human description by comparing semantic similarity in the latent embedding space.Source: §3.3 · p. 5

3.3.2 Task 2: Interactive Scent Comparison

  • Task 2 tests whether the embedding space represents transitions between scents in ways aligned with human comparative descriptions.The example compares a fresh mint scent with a more floral rose scent.Source: §3.3 · p. 5

3.4 The AI Guessing Mechanism

The AI guesses by retrieving the scent embedding most similar to a query vector. Task 1 builds a query from a single-scent description, while Task 2 updates a reference-scent vector using comparative descriptions.

3.4 The AI Guessing Mechanism

  • The guessing mechanism uses cosine similarity to retrieve the scent embedding most similar to the query from the 20 possible scent vectors.The selected embedding’s index identifies the guessed scent.Source: §3.4 · pp. 6–7

3.4.1 Task 1: Scent Description

  • In Task 1, the system compares the query from Section 3.3 with scent embeddings and reports the guessed scent ID audibly and visually.When a guess is wrong, participants may refine their description and prompt another guess until the scent is identified or a predefined limit is reached.Source: §3.4.1 · p. 7

3.4.2 Task 2: Interactive Scent Comparison

  • In Task 2, the system combines the reference-scent embedding with participants’ comparative descriptions to form an updated query vector for predicting the target scent.Participants can provide multiple comparative descriptions, and the system learns a cumulative description before delivering its guess.Source: §3.4.2 · p. 7

3.5 Scent Delivery and Selection

The study uses a scent-delivery device and a selected set of natural scents organized with the Fragrance Wheel. The selection balances variety within and across scent families to avoid making the task too easy or too difficult.

3.5 Scent Delivery and Selection

  • The digital scent-delivery device pumps air through separate scent channels, each loaded with 250 microlitres of scent on a cellulose sponge.Fragrance levels followed previous studies using the same device.Source: §3.5 · p. 7
  • The study selected 20 scents, with five from each Fragrance Wheel family: Floral, Fresh, Woody, and Oriental.Multiple distinct scents within each family were chosen to avoid ceiling effects, while including all four families avoided restricting the task to one family.Source: §3.5 · pp. 7–8
  • The selected fragrances were natural essential oils and extracts, sourced from the same supplier whenever possible to maintain consistency across scent quality and formulation.The lemon scent, for example, came from lemon peel oil extract.Source: §3.5 · p. 8

4 User Study

The study used two interactive scent-description tasks with 40 non-expert participants to evaluate how well the AI interpreted human olfactory descriptions. A repeated-measures design and quantitative ratings plus interviews provided complementary measures of performance and alignment.

4.1 Participants

  • Forty non-expert participants (22 female, 18 male; aged 19–50) took part in the in-person study.They came from 14 countries across four continents and were native or highly proficient English speakers.Source: §4.1 · p. 8

4.2 Study Set-up and Procedure

  • Participants completed prescreening and demographics, then scent-description and scent-comparison tasks, followed by an interview.Scent exposure was standardized at ten seconds, and AI guesses were limited to three per Task 1 round and five per Task 2 round.Source: §4.2 · p. 9, Figure 5 · p. 9
  • In Task 1, participants described single scents and could refine descriptions after incorrect guesses; in Task 2, they described differences between reference and target scents.Task 2 recorded familiarity, intensity, and similarity ratings, with validity and similarity set to 10 when the AI guessed correctly.Source: §4.2 · p. 9

4.3 Evaluating AI Sniff Performance

  • A repeated-measures mixed-methods evaluation used balanced scent assignments and a Latin Square arrangement for Task 2.Task 2 validity and similarity ratings assessed how well guesses matched descriptions and perceived scent characteristics; interviews elicited feedback on performance and future interactions.Source: §4.3 · pp. 9–10

5 Study Results

Across two tasks, the AI identified relatively few target scents and showed marked scent-specific biases. Ratings and interviews also indicated limited alignment, while exploratory GenAI comparisons found that generated scent representations usually differed from human ones.

5.1.1 Overall performance

  • 27.50% of Task 1 rounds and 37.50% of Task 2 rounds were correct; the increase was not statistically significant (z = 1.54, p = 0.0618).Task 1 had 22 correct predictions out of 80 rounds; Task 2 had 60 out of 160.Source: §5.1.1 · p. 10

5.1.2 Scent-specific Performance

  • 100%: Lemon was identified in all Task 1 rounds; Peppermint reached 87.5% in Task 2, while Rosemary and Black pepper were never identified in Task 2.Rosemary was never identified in Task 1 either.Source: Figure 6 · p. 11, §5.1.2 · p. 11

5.1.3 Confusion Matrices

  • Peppermint attracted 30 Task 1 guesses and 137 Task 2 predictions, including misguesses for rosemary, eucalyptus, pine, and black pepper.Gardenia was most commonly guessed as Rose.Source: Figure 7 · p. 11, §5.1.3 · p. 12

5.2 Familiarity Scores

  • Familiarity was not reliably related to success: Task 1 Pearson r = 0.4004, n = 20, p = 0.0802, and Task 2 Pearson r = −0.1577, n = 20, p = 0.5068.Source: §5.2 · p. 12

5.3 Validity Scores for Interactive Scent Comparison (Task 2) and its Relationship to the AI’s Performance

  • 5.30 (SD = 2.98): Task 2 validity ratings were marginally valid overall, but failed guesses were around marginally invalid and did not differ significantly across guesses.The mean was significantly above the neutral midpoint of 5, with a minimal effect size.Source: §5.3 · p. 12, Figure 8 · p. 13

5.4 Similarity Scores for Interactive Scent Comparison (Task 2)

  • Similarity rose from initial similarity (mean = 3.68, SD = 2.57) to the 1st guess (mean = 4.70, SD = 2.52), but not at later guesses.The initial-to-first-guess increase was significant (p = 0.0051); subsequent changes were not (all p > 0.05).Source: Figure 8 · p. 13, §5.4 · p. 13

5.5 Interview results

  • Thirty participants reported lower-than-expected alignment, and 21 said AI guesses often deviated from their descriptions; 19 had difficulty verbalizing smells.Participants cited emotional attachment and personal sensory experiences as factors the AI struggled to capture.Source: §5.5 · p. 14

5.6 Replacing Human Participants with GenAI Models

  • Most GenAI embedding points were far from human scent centroids, with lemon the sole high-alignment case; human word clouds emphasized “sweet,” “fresh,” and “woody,” unlike AI’s “undertone” and “aroma.”The exploratory comparison encoded generated scent descriptions and used t-SNE to visualize human and AI representations in 2D.Source: Figure 9 · p. 15, §5.6 · p. 15, Figure 10 · p. 16

6 Discussion

Sniff AI showed limited but measurable alignment with human scent descriptions, with performance varying by task and scent. The discussion links these differences to how people describe smells and to uneven performance across scent categories.

6 Discussion

  • Task 1 achieved 27.50% success and Task 2 achieved 37.50%, indicating limited alignment despite improvement in the comparison task.The authors note that Task 2 improved across all scent families, particularly Floral, while the specific reason for that family’s gain remains unclear.Source: §6 · p. 16, §6.2 · p. 17

6.1 AI’s Performance in Understanding Scents

  • Task 1 succeeded on 27.50% of scent descriptions; most scents received one correct guess or none, and two scent pairs were each confused five times.The frequent confusions were eucalyptus with peppermint and gardenia with rose.Source: §6.1 · p. 16
  • Scent descriptions were difficult to verbalize: 19 participants reported difficulty, often using sensory terms or personal associations that may challenge scent matching.The discussion gives examples of descriptions that the AI mapped to a different scent, including rose descriptions guessed as geranium or gardenia.Source: §6.1 · p. 16
  • AI-generated scent descriptions used more abstract, expert terminology than everyday speech, which may affect everyday human-AI communication.The authors suggest this difference may reflect pretraining on professional, formal, or technical data.Source: §6.1 · pp. 16–17
  • Human- and AI-generated descriptions separated in the t-SNE visualization, while lemon was a stated exception with 100% identification and closely clustered descriptions.The discussion notes that descriptions vary more for scents than for tactile qualities, which may contribute to less consistent performance across models.Source: §6.1 · p. 17

6.2 Limited Alignment in Smell Experiences

  • Task 2 reached 37.50% success and improved across all scent families, with the largest gain in Floral; reference points appeared to help participants describe scent differences.The reason for Floral’s marked improvement remains undetermined.Source: §6.2 · p. 17
  • A prediction could be rated highly valid but dissimilar, distinguishing interpretation of a description from matching the target scent’s perceptual profile.Across rounds, similarity rose initially and then fell between the first and second guesses, with minimal, nonsignificant iterative improvement.Source: §6.2 · p. 17

6.3 AI Exhibits a Bias toward Specific Scents and Particular Scent Characteristics

  • Performance varied across scent families and familiarity was not significantly correlated with success; the discussion reports accuracy of 25% for familiar Lavender and 50% for least-familiar Black Pepper.The AI often defaulted to peppermint or lemon for related descriptors, while rosemary had a 0% success rate in both tasks; peppermint was also confused with rosemary and eucalyptus.Source: §6.3 · pp. 17–18

7 Limitations and future work

The study’s conclusions are bounded by its limited scent sample and non-expert participants. The authors propose broader scent-language training data and multimodal approaches as future directions.

7 Limitations and future work

  • The scent sample was limited and based on the Fragrance Wheel, constraining coverage of scent diversity.The authors propose extending the selection and collecting more descriptions through in-person studies and online surveys.Source: §7 · p. 18
  • The study primarily recruited non-experts, so its conclusions reflect the general population and may not generalize to professionals.The authors suggest comparing experts and non-experts under the same settings.Source: §7 · p. 18
  • The authors identify a need for training data with more diverse scent descriptors and emotional expressions, alongside possible multimodal inputs combining chemical analysis and human descriptions.They also propose Human-in-the-Loop approaches such as RLHF for adapting predictions through user feedback.Source: §7 · p. 18

8 Conclusion

The paper presents Sniff AI, an LLM-embedding-based system, and evaluates its alignment with human smell descriptions in a 40-participant study. The system captures some scent semantics but has limited accuracy and biases toward certain scents.

8 Conclusion

  • Sniff AI uses an LLM embedding model to recognize voice input, predict scents, and deliver them in real time.Forty participants sniffed scents, described them, and interacted with the system during the study.Source: §8 · p. 19
  • The study found that the embedding space captures scent-related semantics, but accuracy is limited and performance is biased toward certain scents.Participants also described the interaction as fun and interesting.Source: §8 · p. 19
  • The authors identify potential scent-focused AI uses in entertainment, healthcare, and security, while noting difficulty with nuanced or subjective scent descriptions.Examples include perfume selection, support for people with olfactory impairments, and substance detection.Source: §8 · p. 19

Related papers

Loading 2411.06950v1…