Source-linked AI summary
Investigating Human Priors for Playing Video Games
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L. Griffiths, Alexei A. Efros
TL;DR
The paper investigates why humans solve sparse-reward video games efficiently and how different human priors contribute to that ability. It systematically masks visual information in controlled game variants and finds that removing some priors drastically slows human solving, while general object and visual-consistency priors are especially important.
Problem
Deep RL agents can require millions of action inputs on simple games, while humans use prior knowledge about physics, semantics, and affordances; the paper asks how important these priors are for solving video games.
Method
The authors conduct human ablation studies in a specially designed game environment by systematically masking visual information associated with different priors.
Results
Removing some priors increased human solving time from 2 minutes to over 20 minutes, with general priors about objects and visual consistency more critical than specific gameplay knowledge.
Takeaways & Limitations
The results demonstrate that prior knowledge helps humans explore sparse-reward environments efficiently and may inform efforts to build more human-like, sample-efficient RL agents.
Takeaways & Limitations
The study primarily investigates object priors and some physics priors, while other human priors about psychology and general video-game conventions remain future work.
Abstract
from arXiv · showhide
What makes humans so good at solving seemingly complex video games? Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors on human performance. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play. Videos and the game manipulations are available at https://rach0012.github.io/humanRL_website/
1. Introduction
The paper asks why humans solve sparse-reward video games efficiently despite relying on far fewer actions than deep RL agents. Through controlled game modifications, it quantifies how different human priors affect performance and finds that removing some priors severely slows gameplay.
- Humans took more than twice as long and used about 6500 action inputs when semantic and affordance cues were removed, compared with about 2 minutes and 3000 inputs originally.The modified game was structurally identical but harder to interpret and navigate.
- A curiosity-based RL agent used about 4 million action inputs on both game renderings, unlike humans whose performance worsened after re-rendering.Standard A3C was too sample-inefficient for the sparse-reward setting.
- The study systematically masks visual information in a controlled game environment to quantify the importance of semantic, object, affordance, and visual-consistency priors.The game was designed to be sufficiently difficult for performance differences between conditions to be measurable.
- Removing some prior knowledge increased human completion time from 2 minutes to over 20 minutes, showing that priors strongly affect gameplay efficiency.
- General priors about object importance and visual consistency were more critical for fast solving than specific knowledge such as climbing ladders or using keys to open doors.
2. Method
The authors built a browser-based platform game with sparse terminal rewards and created structurally identical variants that mask visual or physical cues. Human performance was measured across these manipulated versions under controlled participation conditions.
- The browser game requires collecting a key and reaching a door while avoiding enemies, spikes, and falls, with a terminal reward of +1 for success.Arrow keys control the agent, and failures reset the game.
- Game variants re-render entities and alter physical properties to mask priors, while preserving the underlying structure, reward, and shortest path to the goal.
- Human performance was measured with 120 participants instructed to finish as quickly as possible without being told the goals or reward structure.Each participant played once, with a 30-minute maximum and payment for successful completion.
- The manipulations target semantics, object concepts, affordances, similarity, ladder interactions, and gravity through masked, recolored, modified, or rotated game elements.
3. Quantifying the importance of object priors
Human performance declines as visual priors are masked, with object identity and visual consistency emerging as especially important guides for efficient exploration.
- 3.1. Semantics: Without semantics, only 42 of 120 participants reached the key before the door, and reaching the door after obtaining the key took significantly longer.The normalized timing analysis was intended to separate this effect from the overall longer completion time.
- 3.1. Semantics: 6.1 minutes, 13.7 deaths, and 9,400 explored states followed reversed object semantics, making gameplay harder than both the original and masked-semantics versions.Participants took longer when familiar positive and negative meanings were assigned to the wrong entities.
- 3.2. Objects as Sub-goals for Exploration: 7.7 minutes, 20.2 deaths, and 12,232 explored states followed removal of the concept of objects, nearly four times the original completion time.Performance was significantly worse than when only semantic information was removed, suggesting that objects guide exploration toward useful sub-goals.
- 3.3. Affordances: 4.7 minutes, 10.7 deaths, and 7,031 explored states resulted from masking affordances, with performance statistically indistinguishable from the masked-semantics condition.The manipulation preserved visible objects and semantics but obscured which textures represented free space, platforms, and ladders.
- 3.4. Things that look similar, behave similarly: 7.6 minutes and 11,715 explored states occurred without visual similarity, while deaths remained lower than in the no-object-information condition.The authors identify visual similarity as the second most important prior after directing exploration toward objects.
4. Taxonomy of object priors
Masking all investigated object priors makes the game extremely difficult, while the proposed taxonomy links earlier-developed priors to greater importance in gameplay.
- 4. Taxonomy of object priors: 20 minutes: simultaneously masking information about objects, semantics, affordances, and visual similarity made the game extremely hard for humans.Participants reported that they could solve it only by memorizing the game.
- 4. Taxonomy of object priors: Preserving physics and motor-control priors did not prevent severe difficulty when visual interpretation became unusable.The authors interpret this manipulation as approximating the lack of prior knowledge available to reinforcement-learning agents.
- 4. Taxonomy of object priors: The authors suggest that incorporating human-like prior knowledge could improve artificial-agent performance.This motivation follows from the difficulty humans experienced when the game was rendered to prevent use of visual interpretation priors.
- 4. Taxonomy of object priors: The proposed taxonomy ranks object priors by developmental order and their criticality for human problem solving in video games.The paper relates infant development from primitive object notions toward visual similarity, semantics, and affordances to the importance of these priors in gameplay.
5. Physics and motor control priors
The experiments show that reversing gravity or reversing key controls substantially impairs human gameplay, indicating that both physical and motor-control priors support efficient solving.
- 5. Physics and motor control priors: Close to 3 minutes: reversing gravity significantly increased participants’ solution time, deaths, and explored states relative to the original game.The time, death count, and state-count differences were each significant at p < 0.01.
- 5. Physics and motor control priors: Close to 3 minutes: reversing arrow-key controls also made participants take longer to solve the game, with significantly more deaths and explored states.The manipulation mapped each key to the opposite movement or vertical action.
- 5. Physics and motor control priors: Gravity reversal and key-control reversal produced similar participant performance, with no significant difference between conditions.Both manipulations disrupt intuitive expectations about the consequences of actions or physical motion.
6. Controlling for change in complexity
RL performance was largely unchanged when semantics, objects, or affordances were masked, supporting the interpretation that human performance losses reflected removed priors rather than visual complexity alone.
- 6. Controlling for change in complexity: The control analysis was designed to test whether altered visual complexity, rather than masked prior knowledge, explained changes in human performance.The human and RL experiments used similar manipulations, although the RL game was smaller because the larger game was too hard for current agents.
- 6. Controlling for change in complexity: The RL agent showed no significant mean-score difference after removing semantics, object concepts, or affordances from the game.Compared with the original game, these manipulations had p > 0.05.
- 6. Controlling for change in complexity: Nearly twice as many interactions were required when visual similarity was masked, unlike the other RL game manipulations.The authors attribute this worsening to convolutional neural networks implicitly imposing a visual-similarity prior, rather than simply to changed visual complexity.
7. Discussion
The discussion argues that human priors enable efficient exploration in sparse-reward games, while warning that strong priors can also constrain exploration and leave important game-playing priors unexamined.
- Human prior knowledge helps people explore sparse-reward environments efficiently, suggesting that RL agents could benefit from mechanisms inspired by human cognition.The authors connect prior knowledge with rapid exploration and learning from successful actions, and propose incorporating reusable knowledge into RL systems.
- The study primarily examines object priors and some physics priors, leaving broader game-playing priors for future investigation.Examples of unexamined priors include expectations that games have goals and that moving upward or rightward signals progress.
- The paper proposes biasing exploration toward visibly distinct entities because humans treat them as interesting sub-goals.This is presented as a concrete direction for improving RL efficiency while incorporating insights from quantified human priors.
- Strong priors can misdirect exploration: humans may pursue presumed goals while missing hidden rewards, producing suboptimal outcomes.Figure 9 presents a hidden-reward game comparing average rewards accumulated by human players and a random agent.