It Takes Two to Tango: Towards Theory of AI's Mind

1. Introduction

• The authors position ToAIM as a complement to research that emphasizes AI modeling human intentions, beliefs, tendencies, contexts, and mental states.They argue that human understanding of AI can help teams decide when its behavior should be trusted.Source: §1 · pp. 1–2
• Theory of AI’s Mind (ToAIM) is the paper’s call for humans to understand AI’s beliefs, knowledge, strengths, weaknesses, and quirks in collaborative settings.The authors argue this intuition matters particularly when AI is imperfect; they do not require laypeople to understand its internal workings.Source: §1 · p. 2
• Human subjects estimate Vicki’s success or failure in Failure Prediction and its exact response in Knowledge Prediction.Failure Prediction measures understanding of Vicki’s strengths and weaknesses; Knowledge Prediction requires a deeper understanding of its behavior.Source: §1 · p. 2
• The study tests whether familiarizing people with Vicki and exposing them to its internal states improves their estimates of its behavior.The estimates concern both Failure Prediction and Knowledge Prediction.Source: §1 · p. 2
• The authors report that people predict Vicki’s successes, failures, and outputs better than chance, and that familiarization with a few examples improves performance.They also report that existing explanation modalities do not enhance prediction of Vicki’s behavior.Source: §1 · p. 2

2. Related Work

• Unlike work in which AI agents model humans, this paper studies how humans can understand AI agents’ quirks, weaknesses, and beliefs.Related work includes agents predicting human motivations, intentions, actions, tendencies, or contexts,, [58],,.Source: §2 · p. 3
• Prior explainable-AI work focuses on transparency and trust, while this paper measures whether explanations improve people’s predictions of AI success, failure, and responses.The related work spans introspective and justification-based, natural-language and visual, and explicit and implicit attention explanations.Source: §2 · p. 3
• Unlike statistical failure-prediction models and studies of human adaptation in interactive tasks, this work measures human modeling of an AI and tests training and explanations.It also measures prediction of the AI’s actual output, not only its failure modes.Source: §2 · p. 3

3. Meet Vicki

• Vicki is an alternating co-attention VQA model trained with VGG-19, with 1,000 possible answers and 62.2% test-standard accuracy.The paper identifies it as the model by Lu et al. [33] and reports human accuracy of 83.3%.Source: §3 · p. 3
• Vicki’s predictable quirks include weak image understanding, reliance on early question words and language priors, poor reasoning and counting, and a fixed answer vocabulary.It also answers questions irrelevant to the image and exhibits dataset-linked biases, such as often answering “yes” to yes/no questions or “white” to color questions [25].Source: §3 · p. 4
• Across image montages, Vicki gives the same response to a given question, and examples reveal recurring visual patterns it may rely on.The paper suggests that exposure to examples of these behaviors may help subjects form an intuition about Vicki.Source: §3 · p. 4

4. Meet the tasks

• Montages show Vicki giving the same response to a given question across images, with common visual patterns apparent within each montage.The figure illustrates recurring behavior that subjects might learn from examples.Source: Figure 2 · p. 4
• Failure Prediction asks people to judge whether Vicki will answer an image question correctly, without showing them Vicki’s predicted answer.The task measures whether people can anticipate Vicki’s success or failure.Source: §4 · p. 4
• Knowledge Prediction asks people to guess Vicki’s exact answer from its 1,000 output labels, testing a deeper understanding of its behavior.Unlike predicting success or failure, it requires guessing the response Vicki will produce for the image and question.Source: §4 · p. 4
• The authors connect Knowledge Prediction to translating an image into what Vicki might say, while noting that testing the reverse translation is future work.They suggest strong performance may correlate with the reverse task, which could help when users cannot directly access visual content.Source: §4 · p. 5

5. Perception of VQA

• In FP, subjects judged whether Vicki would answer correctly; in KP, they predicted its exact response, including on irrelevant image-question pairs.FP ground truth followed the VQA criterion of matching at least 3 of 10 human answers; irrelevance makes correctness ill-defined for FP.Source: §5.1 · p. 5, §5.2 · pp. 5–6

5.1. Failure Prediction (FP)

• 54.24% class-normalized FP accuracy indicates that untrained subjects predicted Vicki’s failures better than chance.Raw accuracy was 59.88%, compared with 61.52% for always guessing success.Source: §5.1 · p. 5
• 75.46% of subjects’ success predictions were optimistic, including for number-answer questions, where VQA models were described as ineffective.Subjects expected Vicki to answer number questions correctly quite often.Source: §5.1 · p. 5

6. Familiarizing people with Vicki

• Feedback exposed Vicki’s answer or correctness after each training response, plus a live score; the test phase provided no further feedback.Training used 50 question-image pairs, followed by 50 test pairs.Source: §6.1 · p. 6, §6 · p. 6

6.1. Does feedback help?

• 29.84% higher mean accuracy in KP with instant feedback than without feedback; FP accuracy was 13.09% higher with feedback.KP with feedback reached 51.11% mean accuracy, and subjects outperformed the most-popular-answer priors.Source: Figure 5 · p. 6, §6.1 · p. 7
• Feedback helped subjects learn Vicki-specific biases, such as defaulting to “no” for rain and relying on coarse scene cues for people counts.For open-ended time questions, feedback helped subjects identify its answer range and correlations with visual patterns.Source: §6.1 · p. 7
• Montage training yielded 68.7% training accuracy but 41.6% test accuracy, below the 51.1% test accuracy with individual-image training.The authors conclude that keeping training and test tasks alike is more effective.Source: §6.1 · p. 7

6.2. Do explanation modalities help?

• Subjects saw top-five prediction confidence, explicit image and question-word attention maps, or Grad-CAM visualizations of implicit attention.The study used three explanation modalities across 84 tasks and collected over 65,000 responses from 415 subjects.Source: §6.2 · pp. 7–8, Figure 6 · p. 8
• An MLP using Vicki’s output softmax reached 81% FP accuracy, versus 60% for a decision stump and 65% for ALERT.Machine methods were trained on about 66% of VQA-val, while subjects trained on 50 examples; the authors report these for context, not as a capability comparison.Source: §6.2 · p. 8
• With instant feedback, explanation modalities produced no statistically significant improvement over feedback alone on either task, and some results were worse.Subjects may overfit to the explanations and hallucinate patterns; Figure 5 summarizes the test accuracies.Source: Figure 5 · p. 6, §6.2 · p. 8

7. Conclusion

• With 50 training examples, lay people got better at predicting Vicki’s responses and failures.The paper proposes Failure Prediction and Knowledge Prediction as measures of understanding a VQA model’s behavior.Source: §7 · p. 8
• The experiments evaluate explanation modalities after instant-feedback familiarization; testing explanations without that feedback remains future work.The authors describe this sequence as the natural progression for real-world familiarization.Source: §7 · p. 8
• Existing explanation modalities did not make Vicki’s failures more predictable and slightly worsened people’s Knowledge Prediction performance.The authors call for improved modalities that help humans predict AI behavior.Source: §7 · p. 9
• The authors report informal evidence that Failure Prediction and Knowledge Prediction performance with one VQA model generalized to another.They envision familiarization whenever a major AI update is deployed.Source: §7 · p. 9
• The study leaves broader testing and the link between prediction skill and team success for future work.Proposed extensions include other models, tasks, explanation modalities, larger human studies, and cooperative games.Source: §7 · p. 9

A. Introduction

• The appendix includes scenario motivation, task demonstrations, further examples of Vicki’s quirks, participant insights, and an AMT survey about public perceptions of AI.Source: A. Introduction · p. 11

B. Visual Recognition Scenarios and Applicability of ToAIM

• Understanding Vicki’s failure modes can help users decide when to trust its answers, but this study focuses on people viewing the image and question while predicting failures and responses.The appendix also notes possible use by blind users and people working with tele-operated robots.Source: B. Visual Recognition Scenarios and… · p. 11

C. Interfaces

• The appendix links to the Failure Prediction and Knowledge Prediction interfaces and demonstration videos, each illustrating a single task setting.Source: C. Interfaces · p. 11

D. Vicki’s Quirks

• Vicki’s answers appear shaped by salient objects and question phrasing; for example, it answers “blue” for grass color even when no grass appears, and “banana” for sign text.In the grass-color examples, it returns the scene’s dominant color; for sign questions, it gives a salient object.Source: D. Vicki’s Quirks · p. 11
• One participant expected Vicki to identify animals correctly 90% of the time unless multiple animals appeared in an image.Source: D. Vicki’s Quirks · p. 12
• A participant guessed that brighter colors might predict success but was unsure about multicolored images; another thought Vicki might choose the brightest color.Source: D. Vicki’s Quirks · p. 12
• Some participants questioned how correctness was determined, including whether attending to the right image region guaranteed a correct answer.The authors report that some participants tried to form mental models while completing the tasks.Source: D. Vicki’s Quirks · pp. 11–12

3. FP + IF + Explanation Modalities

• Participants questioned how Vicki’s correctness was determined, and one said looking at the right spot did not guarantee a correct answer.Source: §3 · p. 12
• Comments suggested perceived limits in counting people, answering time questions, and recognizing some words or scene terms.One participant thought Vicki might know small groups but not huge ones; another said it rounded time answers, while another noted missing words such as “lobby.”Source: §3 · p. 12
• Participants also questioned whether color phase and distance explained Vicki’s metrics, while others described the task as interesting but hard.Source: §3 · p. 12

• One participant said Vicki’s night/day phrasing obscured nuances and reduced their score on the task.Source: §6 · p. 12
• Another participant believed Vicki misidentified some colors and tried to reproduce its responses.Source: §6 · p. 12

• One participant said Vicki chose noon or morning when food appeared, even when the image seemed to show breakfast.Source: §7 · p. 12
• A participant questioned the task’s accuracy despite counting carefully and taking time to assess the pictures.Source: §7 · p. 12
• Another participant found it hard to infer what was being sought because few umbrellas appeared in the pictures.Source: §7 · p. 12
• Subjects commonly said Vicki focused on salient objects, struggled with counting, and answered color questions with the dominant image color.Source: §7 · p. 12

Perception of AI

• Before introducing Vicki, researchers surveyed 321 subjects about demographics, AI familiarity, perceived capabilities, expectations across time horizons, and beliefs about how AI works.Source: E. Perception of AI · pp. 12, 16
• Subjects expected AI to handle translation, driving, and speech recognition today, more sophisticated tasks in three years, and humanlike abilities within ten years.Source: E. Perception of AI · p. 16
• Subjects mostly described AI as programmed systems that use large amounts of information and pattern recognition to pursue predefined goals.Source: E. Perception of AI · p. 16
• The authors suggest that building Theory of AI’s mind skills may help people become more active and effective collaborators, while noting uncertainty about when these capabilities will emerge.Source: E. Perception of AI · p. 16
• 80-90% thought AI could recognize faces and infer mood from social media; 65-70% expected handwriting recognition, creativity, or driving; 96% said it could not read minds, while 62% thought it could surpass the smartest human.Source: E. Perception of AI · p. 16