PaperlayerView exact revision on arXiv

Source-linked AI summary

Gibbs Sampling with People

Peter M. C. Harrison, Raja Marjieh, Federico Adolfi, Pol van Rijn, Manuel Anglada-Tort, Ofer Tchernichovski, Pauline Larrouy-Maestri, Nori Jacoby

arXiv:2008.02595v2q-bio.NCcs.AIcs.CVstat.AP

TL;DR

The paper addresses the limited information and slow exploration of MCMCP for recovering human semantic representations. It introduces slider-based Gibbs Sampling with People, formulates it through utility theory, and shows that it outperforms MCMCP while extending to varied perceptual and high-dimensional domains.

  • Problem

    MCMCP’s binary responses provide little information per trial, and its proposal function creates a trade-off between exploration speed and mode precision.

  • Method

    GSP replaces binary MCMCP choices with continuous slider-based coordinate updates, using a utility-theoretic Gibbs sampler with adjustable aggregation toward optimization.

  • Results

    GSP converged faster and to higher perceptual ratings than MCMCP, while the paper applied it across 25 experiments involving 5,178 participants.

  • Takeaways & Limitations

    GSP provides a framework for studying categorical and continuous semantic representations across visual, auditory, and high-dimensional perceptual spaces.

  • Takeaways & Limitations

    Interpretations depend on the chosen stimulus space and participant group, limiting generalization beyond those sampled populations and dimensions.

Abstract

from arXiv · show

A core problem in cognitive science and machine learning is to understand how humans derive semantic representations from perceptual objects, such as color from an apple, pleasantness from a musical chord, or seriousness from a face. Markov Chain Monte Carlo with People (MCMCP) is a prominent method for studying such representations, in which participants are presented with binary choice trials constructed such that the decisions follow a Markov Chain Monte Carlo acceptance rule. However, while MCMCP has strong asymptotic properties, its binary choice paradigm generates relatively little information per trial, and its local proposal function makes it slow to explore the parameter space and find the modes of the distribution. Here we therefore generalize MCMCP to a continuous-sampling paradigm, where in each iteration the participant uses a slider to continuously manipulate a single stimulus dimension to optimize a given criterion such as 'pleasantness'. We formulate both methods from a utility-theory perspective, and show that the new method can be interpreted as 'Gibbs Sampling with People' (GSP). Further, we introduce an aggregation parameter to the transition step, and show that this parameter can be manipulated to flexibly shift between Gibbs sampling and deterministic optimization. In an initial study, we show GSP clearly outperforming MCMCP; we then show that GSP provides novel and interpretable results in three other domains, namely musical chords, vocal emotions, and faces. We validate these results through large-scale perceptual rating experiments. The final experiments use GSP to navigate the latent space of a state-of-the-art image synthesis network (StyleGAN), a promising approach for applying GSP to high-dimensional perceptual spaces. We conclude by discussing future cognitive applications and ethical implications.

1 Introduction

The paper introduces GSP to address limitations of MCMCP for studying human semantic representations, especially inefficient binary responses and slow exploration. Across 25 experiments with 5,178 participants, the paper applies GSP from low-dimensional stimuli to deep-neural-network parameter spaces.

  • Motivation: Semantic representations connect perceptual inputs to meaningful information used by memory, decision-making, and language.The paper frames semantic representation as a central problem shared by cognitive science and machine learning.
  • Motivation: Exhaustive stimulus evaluation fails for high-dimensional spaces, while hand-constructed stimulus sets depend heavily on prior domain knowledge and constrain exploratory research.These limitations motivate adaptive approaches such as MCMCP and GSP.
  • Motivation: MCMCP studies semantic representations adaptively, but binary responses provide only 1 bit per trial and proposal width creates a speed–precision trade-off.Narrow proposals explore modes slowly, whereas wide proposals reduce mode precision.
  • Contribution: GSP replaces MCMCP’s binary choice with continuous slider manipulation, increasing information per trial and removing proposal-function calibration.The method is formulated through utility theory and can shift toward deterministic optimization through aggregation.
  • Scope: The paper evaluates GSP and MCMCP across four studies spanning visual and auditory domains, from simple low-dimensional stimuli to high-dimensional deep-neural-network spaces.The studies include direct comparisons, hyperparameter controls, and validation experiments.

2 Theory

The theory recasts MCMCP and GSP as utility-based human samplers: MCMCP implements Metropolis-Hastings, whereas slider-based GSP implements Gibbs sampling over stimulus dimensions. An aggregation or noise-control parameter moves sampling between probabilistic exploration and deterministic optimization, while interpretation remains bounded by stimulus-space and participant-group choices.

  • MCMC and MCMCP: MCMCP constructs a Markov chain by proposing a candidate stimulus and using a human choice to implement an acceptance function tied to a target distribution.With a symmetric proposal, the acceptance rule is the Barker function A(x*; x) = π(x*)/(π(x) + π(x*)).
  • Utility theory: A utility formulation assigns each stimulus deterministic utility plus noise, allowing categorical and continuous semantic criteria without requiring utilities to be probability distributions.Noise can reflect sensory, cognitive, and population-level variation.
  • Gibbs Sampling with People: GSP implements Gibbs sampling by having participants adjust one stimulus dimension with a slider while holding the remaining dimensions fixed.The slider response supplies the coordinate update for the Gibbs chain.
  • Gibbs Sampling with People: As slider granularity tends to infinity, GSP samples from p(z) ∝e^γℓ(z), and it can represent either categorical or continuous semantic spaces.For categorical judgments, the stationary distribution is proportional to p^γ(z|C); for continuous criteria, utility need not be probabilistic.
  • Aggregation and optimization: Reducing effective noise γ^-1 concentrates choices around high-utility slider positions, shifting GSP toward coordinate-descent optimization while retaining some noise for exploration.The paper also describes KDE-based aggregation of conditional samples as a practical way to approximate this limit in high-dimensional settings.
  • Interpretation and scope: GSP can use within-participant or across-participant chains, but the paper focuses on group-level representations and warns that findings depend on the defined stimulus space and participant population.For example, results restricted to male voices or US participants may not generalize to other voices or cultures.
  • Related paradigms: Unlike serial reproduction, GSP requires participants to manipulate only one dimension at a time, enabling exploration of richer high-dimensional spaces while providing a probabilistic model for slider judgments.It also differs from related discrete and multidimensional adjustment paradigms in how judgments are solicited and shared across participants.

3 Studies

Across color, emotional prosody, musical chords, and StyleGAN faces, GSP generated perceptually meaningful samples and converged rapidly, while studies also identified important domain-specific scope limits and model biases.

  • 3.1 Color: GSP converged faster and achieved higher ratings than MCMCP for semantic color sampling, with aggregation further improving ratings.The advantage persisted after controls for trial duration, proposal width, question wording, and other design choices.
  • Scope and limitations: Color findings may not generalize globally because color naming varies across cultures, while chord findings are limited by three-tone artificial stimuli and a participant group whose consonance judgments may vary cross-culturally.StyleGAN-derived facial samples can also inherit biases from the model and its FFHQ training data, and results reflect the participant group’s stereotypes.
  • 3.2 Emotional prosody: Emotional-prosody chains produced recognizable feature profiles for sadness, happiness, and anger, while validation ratings rose during the first sweep and then plateaued at a mean contrast of 0.9 points.Observed feature patterns were generally consistent with previous research.
  • 3.3 Musical chords: Musical-chord KDEs revealed integer tuning peaks, dips at the semitone and tritone, and modes corresponding to several Western sonorities; KDE modes outperformed raw samples in validation ratings.The study extends prior work beyond single intervals or fixed tuning systems.
  • 3.4 Faces: In StyleGAN’s face space, chains reached highly rated target samples after one full sweep of the 10 dimensions, with similar success across dimensionality-reduction and aggregation methods.The stimulus space used the top 10 PCA components of StyleGAN’s intermediate latent code, each varying within two standard deviations.

4 Summary and conclusion

The paper concludes that GSP extends semantic-representation methods to richer perceptual spaces by combining continuous human judgments, aggregation, and low-dimensional navigation of generative models.

  • Summary and conclusion: GSP recovered semantic representations across color, emotional prosody, musical chords, and faces, producing rich representations across varied perceptual domains.The method organizes participants into virtual Gibbs samplers that generate stimuli associated with semantic representations.
  • Summary and conclusion: Continuous sliders, no proposal-function tuning, and one-dimension-at-a-time updates are identified as factors that can support rapid convergence on high-quality samples.The slider supplies more information per trial than binary choices, while single-dimension comparisons may be easier for participants.
  • Summary and conclusion: Utility-theoretic formulation extends GSP and MCMCP to continuous and categorical representations, relaxes assumptions about priors and response noise, and makes aggregation practical for slow-to-generate stimuli.Aggregation also averages perceptual noise while increasing the participant-to-stimulus ratio.
  • Summary and conclusion: Applying GSP to deep neural synthesis models requires a relatively low-dimensional basis, for which PCA can sometimes provide a workable representation of the latent space.This approach opens a route to studying semantic representations in high-dimensional perceptual spaces.

Broader Impact

The paper presents GSP as a tool for studying subjective semantic judgments, while emphasizing that its outputs must not be treated as objective truth and require bias analysis and rigorous validation.

  • Broader impact: The method broadens the perceptual spaces and granularity that cognitive scientists can study compared with conventional approaches, including potential cross-cultural applications.Cross-cultural use is presented as a research direction requiring care because exploratory outputs reflect participant populations and stimulus models.
  • Ethical recommendations: GSP characterizes participants’ subjective semantic judgments rather than objective truth, so associations such as glasses with perceived intelligence must not be interpreted causally or socially as facts.The paper warns that such misinterpretations can perpetuate or amplify stereotypes, especially concerning race, ethnicity, and gender.
  • Ethical recommendations: Researchers should analyze, report, and ideally avoid biases in both stimulus design and participant recruitment, especially for complex stimuli such as spoken sentences and StyleGAN images.The paper contrasts these potentially biased spaces with simpler examples such as HSL colors and musical triads.
  • Ethical recommendations: GSP findings should be validated with rigorous hypothesis-driven experiments because deep generative models can pass hidden biases into exploratory studies of naturalistic stimuli.The paper presents validation as essential when combining GSP with models such as StyleGAN.

A.2 GSP

GSP models slider-based human choices as Gibbs sampling under utility-based noisy decisions, while its analysis depends on independent noise and exhaustive slider exploration. The section also identifies correlation and context effects as limitations of the derivation.

  • A.2 GSP: GSP models slider choices with i.i.d. Gumbel utility noise, yielding a multinomial-logit choice probability for each candidate point.The participant selects the alternative with highest utility, and the derivation uses independent Gumbel-distributed noise.
  • A.2 GSP: The GSP derivation can be generalized beyond Gumbel noise by replacing its distribution and cumulative functions, although Gaussian noise produces no closed-form probability.The independent-probit formulation still permits exploration because Gaussian and Gumbel distributions have functionally similar shapes.
  • A.2 GSP: The choice-probability derivation assumes independent noise, but intra-participant and neighboring-point correlations can violate that assumption.Across-participant chains avoid intra-participant correlation, whereas within-participant chains approximate an individual utility function; neighboring-point correlation remains a modeling concern.
  • A.2 GSP: Context effects may alter responses when participants repeatedly encounter stimuli, especially in within-participant chains, although an emotional-prosody comparison found no material difference.Across-participant chains reduce repeated exposure within a chain, and the reported test suggests memory effects were not significant in that paradigm.
  • A.2 GSP: The derivation also assumes participants visit every slider position, so incomplete exploration effectively reduces slider granularity.The authors expect limited consequences because participants focus on high-utility regions and can interpolate between visited locations, but identify the assumption as a future-work target.

B.1 Implementation

The experiments used a browser-based PsyNet infrastructure, recruited screened adult Mechanical Turk participants, and applied pre-screening and incentives to improve data quality. The supplementary materials document participant composition and the main and validation experiment tables.

  • B.1 Implementation: All experiments ran in PsyNet on Dallinger, with browser clients communicating with a Python server cluster managed largely through Heroku.The infrastructure organized experiment timelines and supported complex GSP and MCMCP paradigms.
  • B.1 Implementation: Participants were adults recruited from Amazon Mechanical Turk with at least a 95% prior-task approval rate and informed consent.The protocol was approved by the Max Planck Society Ethics Council.
  • B.1 Implementation: Participants were paid at a US $9/hour rate according to completed experiment time, including proportional payment when they exited after failing a pre-screening task.Recruitment and chain completion thresholds were managed automatically, with separate procedures for across-participant chains, within-participant sessions, and rating experiments.
  • B.1 Implementation: The study included 5,178 participants across 25 experiments, with main and validation groups comprising 2,967 and 2,211 participants, respectively.The reported participant ages ranged from 18 to 89 for demographic respondents, and the two groups had similar demographic compositions.
  • B.1 Implementation: Data-quality procedures combined color-blindness, color-vocabulary, and audio pre-screening tasks, with some experiments adding performance-based financial incentives.The pre-screening tasks targeted color perception, English color vocabulary, headphone use, hearing, and resistance to automated responding.

C.1 Supplementary methods

The color studies used HSL stimuli and compared slider-based GSP with pairwise MCMCP across chains, aggregation settings, proposal widths, question types, and validation ratings. Generated samples were evaluated through additional perceptual ratings and exploratory variants.

  • C.1 Supplementary methods: GSP and MCMCP were applied to eight color concepts in HSL space, chosen because it is generally considered to better reflect perceived color relationships than RGB.Each color was encoded by integer hue, saturation, and lightness values.
  • C.1 Supplementary methods: MCMCP used a Gaussian proposal with standard deviation 30 and modulo handling at stimulus-space boundaries, whereas GSP participants directly manipulated a slider.The GSP interface continuously updated the displayed color as participants moved the slider; MCMCP presented color pairs for binary choice.
  • C.1 Supplementary methods: The main chain design used five across-participant chains per adjective, each containing 30 states beyond the initial random state, with each participant contributing up to 40 trials per method.The aggregated GSP condition combined 10 trials per Gibbs iteration using arithmetic means for saturation and lightness and a circular mean for hue.
  • C.1 Supplementary methods: Follow-up experiments varied MCMCP proposal widths, tested three utility questions, evaluated tenfold MCMCP aggregation, and sampled 4,000 uniformly distributed HSL colors.These studies were paired with validation procedures that collected at least five ratings per generated sample while preventing repeated ratings of the same stimulus.

C.2 Supplementary results

Supplementary analyses found that GSP’s advantage over MCMCP persisted across timing, proposal-width, question-type, aggregation, averaging, and utility-distribution checks, although performance depended on aggregation over iterations.

  • Aggregation and timing: Non-aggregated GSP performed better early, whereas aggregated GSP converged more precisely later.The authors attribute this pattern to rapid escape from poor starting states followed by more precise convergence under aggregation.
  • Proposal width: Proposal widths produced little overall quality difference: width 10 made local adjustments, wide proposals explored quickly, and width 30 balanced both behaviors.Narrow proposals often failed to reach the appropriate category, whereas wide proposals struggled with subtle refinements.
  • Aggregation comparison: Aggregated MCMCP slightly improved over non-aggregated MCMCP, but remained far behind the difference between GSP variants.The authors link this limit to MCMCP’s one-bit response information per iteration.
  • Sample averaging: Averaging samples improved MCMCP and non-aggregated GSP but did not resolve MCMCP’s poorly located category colors.Failures remained for chocolate, eggshell, and lavender, among other categories.
  • Utility distributions: MCMCP’s poor performance reflected mislocated secondary peaks rather than merely broader distributions, while aggregation sharpened GSP peaks.Examples included red for “grass,” orange for “lavender,” blue for “lemon,” and unrelated-looking MCMCP samples.
  • Overall comparison: GSP outperformed MCMCP for the color-estimation task despite analyses addressing six possible confounds.The authors conclude that none of these considerations substantially changed the main comparison, while noting that other perceptual domains may differ.

D.3 Supplementary results

Supplementary prosody analyses found similar feature values and validation scores for within- and across-participant chains, with most feature development occurring during the first sweep. GSP also revealed emotion-specific and shared correlations among prosodic features.

  • Chain comparison: Within- and across-participant chains produced broadly similar feature values and validation contrast scores.These results suggest that memory effects did not substantively contaminate within-participant chains.
  • Convergence: Most feature development occurred during iterations 1–7, after which within-participant feature values remained broadly similar.The supplementary material also provides final-iteration audio examples for sad, happy, and angry speech.
  • Higher-order structure: Duration and F0 perturbation correlated for sadness (r = .28) but not anger (r = −.03) or happiness (r = .00), while pitch level and slope correlated across all emotions.These interactions extend analysis beyond independent feature contributions.

E.1 Supplementary methods

The musical-chord study represented triads by two pitch intervals and used GSP to sample pleasantness continuously, validating raw samples and KDE modes across chain iterations.

  • Stimulus representation: Each triad was represented by two semitone intervals from its bass tone, capturing the relative nature of human pitch perception.The study focused on three-tone chords and used the pitch-chord-type representation from.
  • GSP procedure: Participants adjusted a slider to make each chord match “pleasant,” with the current interval pair played when the slider was released.Participants completed three training examples before the main experiment.
  • Chain construction: The main experiment formed 50 chains of 40 iterations, beginning from uniformly sampled interval pairs in [0.5, 11].Each participant contributed up to 20 trials to the chains.
  • Aggregation and symmetry: Data were aggregated continuously and folded across x = y because swapping the two intervals leaves the generated chord unchanged.This symmetry was used to improve estimated mode quality.

E.2 Supplementary results

Supplementary chord results showed sampling-like GSP trajectories rather than optimization dynamics, with strong marginal modes at the perfect fifth and major third and additional structure matching Western tonal hierarchy.

  • Sampling dynamics: GSP trajectories made large leaps rather than small, converging updates expected from an optimization regime.The trajectory therefore illustrated the sampler’s exploratory behavior.
  • Chord preferences: Strong marginal modes emerged at the perfect fifth (7) and major third (4), with additional peaks at integers and dips at the semitone (1) and tritone (6).The pattern reflected the standard Western tonal hierarchy.

F.1 Supplementary methods

The supplementary methods detail StyleGAN-based GSP experiments, validation studies, alternative aggregation and basis constructions, dataset changes, participant grouping, and bias evaluations.

  • StyleGAN implementation: The StyleGAN study manipulated principal components of the latent representation with an interactive slider, generating stimuli asynchronously to sustain participant throughput.The API varied one active component across specified standard deviations, while pre-generation reduced waiting during trials.
  • Main experiment: The main experiment evaluated six adjectives using 18 across-participant chains, each with 50 iterations plus an initial random state and five responses per step.Participants were US residents recruited through AMT and completed practice trials before contributing chain responses.
  • Validation: Generated samples were validated by collecting at least 50 ratings per sample across iterations 1–10, 20, 30, 40, and 50.Each validation participant contributed 80 ratings without rating the same sample twice.
  • Follow-up experiments: Follow-up experiments tested KDE-mode aggregation, sparse PCA, ICA, later PCA dimensions, WikiArt training data, global recruitment, and gender-separated dating preferences.Additional rating experiments evaluated the KDE, global-participant, basis-construction, and modeling-pipeline conditions.
  • Bias evaluation: Bias evaluations measured perceived gender, ethnicity, smiling, hats, formal clothing, glasses, and age, using categorical or binary judgments plus age estimates.The selected attributes targeted real-world bias or readily quantifiable visual features, while the authors acknowledged that some cannot be determined definitively from images.

F.2 Supplementary results

Supplementary results show rapid early convergence across StyleGAN experiments, robustness to several methodological variations, and interpretable but dataset-dependent biases in generated face samples.

  • StyleGAN convergence: GSP chains reached highly rated target-category samples after one full sweep of StyleGAN’s 10 dimensions, demonstrating rapid navigation of the generative space.Follow-up experiments reported similar success with alternative dimensionality-reduction and aggregation methods.
  • Sampling behavior: Subsequent samples retained moderate variety after early progress, leaving the process partly stochastic rather than becoming deterministic optimization.The raw-sample trajectories showed that resemblance often stopped improving noticeably after 10 iterations without collapsing to a single static image.
  • Validation results: Most improvement occurred within the first 10 iterations, while trustworthiness remained difficult to capture and KDE peak-picking showed no advantage over arithmetic averaging.These patterns replicated across mean aggregation with US participants, KDE aggregation with US participants, and KDE aggregation with global participants.
  • Basis construction: PCA had an early advantage over sparse PCA and ICA, whereas PCA components 71–80 performed poorly because they contributed little perceptually.The differences between the first three basis methods diminished after more iterations.
  • Bias analyses: The modeling pipeline amplified age–gender associations and inherited visual biases that could steer ‘attractive’ and ‘intelligent’ samples toward women or men.Formal clothing and glasses were associated with perceived intelligence, while smiling was associated with perceived attractiveness and fun.
  • Training dataset: Changing the training dataset to WikiArt produced artistically distinct samples while preserving successful navigation toward subjectively matching target adjectives.This demonstrates both the method’s portability across generative spaces and the dependence of outputs on training data.
  • Conclusion: The authors conclude that GSP effectively explores StyleGAN’s generative space, but dataset bias must be addressed when interpreting its outcomes.They recommend studying generative-model biases and developing less biased training datasets for future cognitive applications.
Loading 2008.02595v2…