Dimensions of Disagreement: Unpacking Divergence and Misalignment in Cognitive Science and Artificial Intelligence

Science and Artificial Intelligence

• Disagreement can involve both divergent judgments and misaligned representations, a dimension AI and computational cognitive science help formalize with implications for collaboration and conflict resolution.Divergence-focused accounts capture differences in evaluations but miss differences in the representations and processes that generate judgments.Source: Science and Artificial Intelligence · pp. 3, 4, 6
• A Bayesian account characterizes divergence as the difference between agents’ credences about whether the sun is the center of the world.In the Galileo–Urban example, this is expressed as |PG(S) - PU(S)|.Source: Science and Artificial Intelligence · p. 3
• Early heliocentric and geocentric models made similar predictions about planetary positions despite relying on different representations and explanations of the universe.Their credence differences about whether the sun is central can be characterized as divergence, but the models’ distinct structures are not captured by that measure alone.Source: Science and Artificial Intelligence · p. 4
• Representational alignment concerns the relative structure of representations, so agents may experience individual stimuli differently yet remain aligned if their similarity relations are preserved.The example of color perception is practically unproblematic when both agents agree that red is more similar to purple than to green.Source: Science and Artificial Intelligence · p. 5
• Representational misalignment can be measured by comparing the similarity structure of agents’ representations, including pairwise distances between neural-network activation vectors.Pairwise similarity judgments can also be used to infer which stimuli are closer in representational space.Source: Science and Artificial Intelligence · pp. 5–6
• Providing common data is often proposed to reduce disagreement, but representational misalignment can make polarization common and rational.This occurs when agents differ in their represented hypotheses or in the conditional dependencies they consider.Source: Science and Artificial Intelligence · pp. 6–7
• Different hypothesis sets or dependencies can produce divergent conclusions from the same data; data may help resolve disagreement when agents’ representations predict different observations.The same stellar observation can support both geocentric and heliocentric views when each model explains it differently.Source: Science and Artificial Intelligence · p. 7

Enriching Extant Research

• Misalignment offers a distinct, synergistic explanation for biased advice-taking: people may infer an advisor’s representation from estimates and discount advice after errors when it seems complex and misaligned.In a simple estimation task, sampling an advisor’s estimates across predictor values could reveal their representation of the relationship between predictor and outcome.Source: Implications of the Distinction for… · p. 8
• Because misalignment requires systematic judgments or internal-representation simulation, people may rely on divergence heuristics; alignment inference should be easier across familiar agents than unfamiliar-domain artificial agents.People may use their own representations as informative priors in familiar domains, whereas unfamiliar domains such as protein folding pose tougher challenges.Source: Implications of the Distinction for… · p. 9
• In groups, coarse clustering of agents and invariant within-cluster representations may approximate misalignment, but caricatured inferences can contribute to stereotypes.People may infer group disagreement from estimate variance, while coarser representations risk oversimplifying individuals’ models.Source: Implications of the Distinction for… · p. 10
• Misalignment reframes persistence amid dissent: subjectivity may reflect expectations of representational diversity, potentially making some domains less amenable to conciliation than formally uniform ones.Abstract domains may invite expectations that disagreement is incommensurable, while uniform domains such as games, markets, or mathematics may be more conducive to conciliation.Source: Implications of the Distinction for… · p. 10
• Marti et al. (2023) analyses suggest that 10-30 variants of common concepts for even simple categories exist in the population, despite people assuming shared semantics.The analyses found substantial variation in word representations that people could not introspectively access.Source: Implications of the Distinction for… · p. 11
• Misalignment in word meanings predicts communication failures, motivating research on how perceived and actual misalignment jointly shape conflict resolution and collaboration.Recent computational cognitive science work lays groundwork for studying these effects across influential domains.Source: Implications of the Distinction for… · p. 11

• RLHF trains models to prioritize outputs agreeing with feedback, risking surface alignment that hides harmful biases and undetectable misalignment (Bai et al., 2022).Source: Implications of the Distinction for AI… · p. 12
• Because divergence can enable alignment and progress, LLM research should train models to express divergence for misalignment detection while retaining usability.Source: Implications of the Distinction for AI… · p. 12
• Supervised learning can improve convergence with human-generated labels while reducing representational alignment (Muttenthaler et al., 2022), and performance-alignment may be domain-dependent and U-shaped (Sucholutsky & Griffiths, 2023).Source: Implications of the Distinction for AI… · p. 13
• Human-model alignment is best predicted by training data and objective, whereas model scale and architecture have essentially no effect (Muttenthaler et al., 2022).Source: Implications of the Distinction for AI… · p. 13
• Divergence and misalignment are correlated but distinct: key divergences can precipitate misalignment, but neither is necessary nor sufficient for the other.Source: Implications of the Distinction for AI… · p. 13
• Across diverse agents, misalignment—driven by differences in data, cognitive architectures, and goals—precipitates divergence.Source: Implications of the Distinction for AI… · p. 13
• When collaborators pursue different subgoals and encounter different data, representations may diverge and hinder communication.Source: Implications of the Distinction for AI… · p. 13
• Promising alignment policies include periodically sharing informative datapoints or prototypes and assigning teammates other goals.Sharing can involve selecting and sequencing informative judgments in a recipient-sensitive way.Source: Implications of the Distinction for AI… · pp. 13–14

• Disagreement arises from a complex interplay between divergence and misalignment, although past judgment and decision-making research has largely focused on divergence.Source: Conclusion · p. 14
• Recent AI research has developed efficient methods for measuring and comparing agents’ representations, with potential to enrich judgment and decision-making research and beyond.Source: Conclusion · p. 14
• Misalignment may help explain biased advice-taking, persistent controversial beliefs, and algorithm aversion.Source: Conclusion · p. 14
• Open questions remain about how to infer and resolve misalignment in humans, artificial agents, and groups.Source: Conclusion · p. 14
• Deeper understanding can help move beyond suppressing divergence and leverage diverse perspectives to solve difficult societal problems, with dissent supporting productive diversity.Source: Conclusion · p. 14