Source-linked AI summary
Toward a Metrology for Artificial Intelligence: Hidden-Rule Environments and Reinforcement Learning
Christo Mathew, Wentian Wang, Jacob Feldman, Lazaros K. Gallos, Paul B. Kantor, Vladimir Menkov, Hao Wang
TL;DR
Researchers lack a way to identify what AI systems learn from training, limiting understanding of how they work and where their limits lie. The paper compares Feature-Centric and Object-Centric state representations in GOHR using a Transformer-based Advantage Actor-Critic framework. The proportion of “not different” outcomes rose from 66% with 5 runs to 83% with 15 runs; 29 out of 30 color-vs-shape comparisons showed no significant difference.
Problem
Researchers lack a way to identify what AI systems learn from training, limiting understanding of how they work and where their limits lie.
Source: §1 · pp. 1–2Method
The paper compares Feature-Centric and Object-Centric state representations in GOHR using a Transformer-based Advantage Actor-Critic framework.
Source: §6 · p. 23Results
The proportion of “not different” outcomes rose from 66% with 5 runs to 83% with 15 runs; 29 out of 30 color-vs-shape comparisons showed no significant difference.
Source: §4.4.4 · p. 22Takeaways & Limitations
Laboratories combining computer science and psychology have the potential to lay foundations for a new science of machine intelligence.
Source: §1 · p. 2, §6 · p. 25Hidden-rule tasks can have many kinds of complexity, so their difficulty may not be organized on a single dimension.
Source: §1 · p. 4
Abstract
from arXiv · showhide
We investigate reinforcement learning in the Game Of Hidden Rules (GOHR) environment, a complex puzzle in which an agent must infer and execute hidden rules to clear a 6$\times$6 board by placing game pieces into buckets. We explore two state representation strategies, namely Feature-Centric (FC) and Object-Centric (OC), and employ a Transformer-based Advantage Actor-Critic (A2C) algorithm for training. The agent has access only to partial observations and must simultaneously infer the governing rule and learn the optimal policy through experience. We evaluate our models across multiple rule-based and trial-list-based experimental setups, analyzing transfer effects and the impact of representation on learning efficiency.
1 Introduction
The paper proposes using controlled psychological-style tasks to probe what AI systems learn and to develop ways of measuring their capabilities. It focuses on hidden-rule tasks, including GOHR, while noting that rule difficulty may have multiple dimensions.
1 Introduction
- Psychology-inspired laboratory instruments are proposed as an initial probe toward a science that clarifies AI capabilities and supports effective collaboration with machine intelligence.The authors suggest that controlled challenges and measures such as solution speed and accuracy can provide fiducial points for studying intelligence.Source: §1 · pp. 2–3
- The nature of AI training remains obscure because systems cannot explain what they know, and correct labels do not reveal what makes an answer correct.The authors argue for a laboratory framework where researchers can know whether training has captured the essence of what is being taught.Source: §1 · p. 2
- Correct answers across repeated examples can provide evidence of rule understanding, while training-example counts and error rates can help estimate task difficulty.The authors propose using the spread around the median number of examples needed to reach a specified error rate to assess how universal a difficulty estimate is.Source: §1 · p. 3
- Hidden-rule tests ask learners to infer which actions are allowed and then learn through repeated trial and error.Repeated variants should prevent success by copying a previous answer; examples can vary object positions while preserving the governing relation.Source: §1 · p. 3
- Because hidden rules can have many kinds of complexity, their difficulty may not be measurable on a single dimension.The authors suggest exploring rule descriptions and action mappings to identify possible fixed points in a space of rule difficulty.Source: §1 · p. 4
- The paper studies GOHR, where agents infer hidden rules from partial observations, and compares Feature-Centric and Object-Centric encodings with Transformer-based A2C.Experiments examine individual-rule learning as well as transfer and generalization from combinations or sequences of rules.Source: §1 · pp. 4–5
2 GOHR Environment
GOHR uses a 6 × 6 board, four buckets, and nine randomly placed pieces; players must infer the hidden rule while clearing the board. The environment provides board and status observations, supports representation-dependent actions, and rewards valid and invalid moves differently.
2 GOHR Environment
- GOHR presents nine randomly selected pieces on a 6 × 6 board with four buckets, and the player must discover the hidden rule while placing every piece in an allowed bucket.An episode ends when all pieces have been successfully placed.Source: §2 · p. 5
2.1 Board State
- The board state is represented as a list of pieces, each with shape, color, and (x, y) coordinates.The board table numbers cells from 1 to 36 and also gives coordinates for their spatial layout.Source: Table 1 · p. 5, §2.1 · p. 6
2.2 Status Codes
- Status codes report whether the last move was accepted or rejected, whether the game is ongoing or finished, and how many moves have been attempted.The agent uses response codes and finish status for evaluation and episode completion; move_count is ignored during training.Source: §2.2 · p. 6
2.3 Action Space
- Actions assign a piece to one of four buckets; FC indexes 36 board positions times four buckets, while OC indexes each object’s four possible bucket assignments.FC actions are one-hot vectors of length 144; OC uses one-hot vectors of length 4 for each object.Source: §2.3 · p. 6
2.4 Reward Function
- Valid placements yield reward 0, while invalid or immovable moves yield −1.The reward is based on the environment’s response code for the attempted placement.Source: §2.4 · p. 7
3 Methodology
Methodology converts GOHR board states into Feature-Centric or Object-Centric inputs, incorporates recent state/action history, and trains Transformer-based A2C over valid actions.
3.1 State Representation and Feature Engineering
- FC encodes each feature as a separate 6 × 6 position map, so the model must learn shape–color associations from positional alignment rather than fused object features.The representation uses eight feature maps; shape and color signals at the same position identify properties of one object.Source: §3.1 · p. 7, Figure 2 · p. 8
3.2 Input and Output Representation
- In the object-centric representation, each board piece has a 20-dimensional vector with one-hot color, shape, and (x, y) position encodings; the input has shape 7 × n × 24, including four action features.The feature-centric input is flattened to dimension 2880.Source: §3.1 · p. 7, §3.2 · p. 8
3.3 Algorithm Implementation
- Transformer-based A2C samples from valid actions and trains its critic with squared error while optimizing the policy with an entropy-regularized objective.The advantage is normalized within a batch for variance reduction, and the entropy coefficient controls exploration.Source: §3.3 · pp. 10–11
3.4 Training and Evaluation Protocol
- Training runs for at most 10,000 episodes and stops early once all three evaluation metrics are achieved.Source: §3.4 · p. 11
3.5 Evaluation Metrics
- The three metrics mark sustained low average error, sustained low maximum error, and a run of successful moves, respectively.Their windows and thresholds define the first episode or move at which each criterion is met.Source: §3.5 · p. 12
- Across 18 rules and both memory windows, the metrics are substantially but not completely consistent, with episode-window measures agreeing especially well on relative rule difficulty.Figure 6 presents the Spearman-correlation heatmap, and Table 3 gives the supporting data.Source: §3.5 · p. 12, Table 3 · p. 12, Figure 6 · p. 13
4 Experiments
The experiments use two GOHR configurations: rule-based runs supply a rule definition, while trial-list runs specify rule sets and initial-board generation and support transfer experiments.
4 Experiments
- Rule-based runs provide the rule definition to the game server; trial-list runs specify rule sets and initial-board generation and also support transfer experiments.Source: §4 · p. 13
4.1 Independent Rule Experiments
Independent rule experiments compare learning across 18 rules and analyze difficulty both by rule properties and by complete rules. Difficulty patterns vary by representation, while category-level ordering across complete rules is more stable than rankings within categories.
4.1 Independent Rule Experiments
- The study trains separate models on 18 rules, then decomposes rules into properties because metric overlap made clear whole-rule rankings difficult.Property comparisons primarily use m⋆, a step-level measure of whether individual moves obey the property; episode-level measures can be noisy when a property applies on few moves.Source: §4.1 · pp. 13–14, §4.1.1 · p. 14
4.1.1 Difficulty Analysis of Rule Properties
- Difficulty tracks abstraction: FC learns positional properties most easily, while feature ordering and conditional properties are hardest; OC shows smaller differences across properties.In OC, feature-to-bucket mapping, all-pieces-of-feature, and quadrant mapping are easiest; reading order and proximity are harder, and bucket ordering is moderately difficult.Source: §4.1.1 · pp. 14–16
- FC learns quadrant mapping, proximity, and reading order most easily, feature-to-bucket mapping and all-pieces-of-feature at moderate difficulty, and feature ordering and conditional properties most slowly.Bucket ordering is also challenging because it depends on an abstract ordering of buckets.Source: §4.1.1 · p. 15
- OC’s easiest properties are feature-to-bucket mapping, all-pieces-of-feature, and quadrant mapping; reading order and proximity are slower, while feature ordering and conditional properties remain hardest.Bucket ordering is slightly more challenging than the easiest properties.Source: §4.1.1 · p. 16
4.1.2 Rule Difficulty Analysis
- Across complete rules, relative difficulty is stable between categories across all three metrics, although the ordering of rules within a category can vary by metric.Quadrant-mapping rules are highly learnable in FC; feature-ordering rules, especially with conditional properties, are most challenging. OC groups a broader set of rules as learnable, while proximity or reading-order combinations are more challenging.Source: §4.1.2 · p. 16, FC Model · pp. 16–17
4.1.3 Key Observations
- FC rule rankings show more metric-driven crossings, while OC has fewer crossings and more coherent difficulty ordering across measures.Crossings cluster within or between related categories; FC’s positional easiest and feature-ordering/conditional hardest properties tend to retain their extremes.Source: §4.1.3 · p. 17
4.2 Transfer Effect Experiments
Transfer experiments test whether prior training on component rules affects learning their combinations. Full-component pretraining accelerates compound-rule learning, whereas partial, extra-rule, and unrelated pretraining yield weaker results.
4.2 Transfer Effect Experiments
- The protocols compare sequential transfer, which trains A and B separately before A+B and then C+D, with partial transfer, which pairs a previously learned rule with a novel one.The experiments use trial-list combinations designed to assess whether prior exposure to component rules facilitates learning combinations.Source: §4.2 · p. 19
4.2.1 Key Observations
- Compound rules learn substantially faster when trained immediately after pretraining on both component rules.The clearest case is training A+B after A → B, which leads to quick convergence.Source: §4.2.1 · p. 19
- Pretraining on only one component with an unrelated rule yields slower convergence than full-component pretraining, and outcomes depend strongly on rule identity and order.For OC rule cw_qn2, ordL1 → quadNearby converges much faster than other listed partial-pretraining sequences, nearly matching cw → quadNearby.Source: §4.2.1 · p. 19
- Adding a third rule after training on both components substantially degrades compound-rule convergence, producing an effect like partial pretraining.Source: §4.2.1 · p. 20
- Pretraining on unrelated rules provides the weakest transfer, with the slowest convergence and minimal benefit.Source: §4.2.1 · p. 20
4.3 Generalization Analysis
Generalization is tested by training on checkerboard positions and evaluating with pieces allowed across the board. OC models generally show lower test-error ratios than FC models, with location-dependent rules remaining difficult.
4.3 Generalization Analysis
- Training restricts pieces to white checkerboard squares, while testing allows positions across the board, including positions not used in training.Source: §4.3 · p. 20, Figure 8 · p. 20
- FC test-error ratios are usually 50–75%, reach 90% for cm_RBKY and sm_csqt, and reach 99–100% for quadNearby and quadMixed1; OC ratios are usually 49–55%, and 65% for those latter rules.Source: §4.3 · p. 20
- The high errors for quadNearby and quadMixed1 are attributed to reliance on precise locations: raw (x, y) encoding makes unseen coordinates reduce performance.Overall, the authors conclude OC generalizes more strongly by associating behavior with object-level features, while FC tends toward memorizing positional patterns.Source: §4.3 · p. 20
4.4 Similarity of shape and color rules
The section tests whether shape-based and color-based rules yield similar outcomes and whether conclusions stabilize as run counts increase.
4.4 Similarity of shape and color rules
- Kruskal–Wallis tests compared expected color- and shape-rule similarities using m⋆, e⋆ mean, and e⋆ max.The rules were expected to be similar because the paired rules differ only in using shape rather than color.Source: §4.4 · p. 21
4.4.1 Comparison within the rule
- For every rule, at least one of the three metrics found the five runs statistically indistinguishable.The test failed to reject the null hypothesis for at least one metric per rule.Source: §4.4.1 · p. 21
4.4.2 Comparison between color and shape rules
- 29 of 30 pairwise comparisons had p-values above 0.05; only FC m⋆ for (col1Ord_BRKY, sha1Ord_qcts) differed (p = 0.036).The comparison aggregated 35 runs per rule.Source: §4.4.2 · p. 21
4.4.3 Effect of increasing the number of runs
- The “not different” share rose from 66% (40/60 comparisons) with 5 runs to 83% (50/60 comparisons) with 15 runs.The authors present the 15-run analysis as a robustness check; the 5-run analysis remains a valid baseline.Source: §4.4.3 · p. 22
4.4.4 Summary
- The summary reports that expected color–shape pairs were mostly not pairwise distinct, while increasing run counts improved the share of indistinguishable outcomes.The summary gives 29 of 30 comparisons with no significant difference for the color–shape analysis and an increase from 66% to 83% for the run-count analysis.Source: §4.4.4 · p. 22
5 Multi-Dimensional Scaling
The section uses pairwise rule comparisons to explore how difficulty relationships among rules can be represented geometrically.
5 Multi-Dimensional Scaling
- The authors used Mann–Whitney p-values for pairs among 18 rule sets, then defined dissimilarity as 1 − p_MannWhitney(i, j) for multidimensional scaling.P-values near 1 indicate sets that could come from the same distribution; values near zero indicate that this is unlikely under the same-distribution assumption.Source: §5 · p. 22
- The first two MDS dimensions show several clusters, but each appears to contain too many different rules.Figure 9 presents the first two principal dimensions of the embedding.Source: §5 · p. 23, Figure 9 · p. 23
- Adding the third dimension resolves the large cluster; whether the clusters reflect algorithmic and rule concepts remains for future work.Figure 10 shows the view using the third dimension as the vertical axis.Source: §5 · p. 23
6 Conclusion
The conclusion frames hidden-rule learning as a possible basis for measuring AI difficulty. Initial findings suggest that task difficulty may be organized geometrically, but measured relationships can differ by metric.
6 Conclusion
- Initial findings suggest tasks may occupy a space where proximity reflects AI difficulty, while different difficulty measures can yield different geometric or ordinal relationships.The proposed tasks include discovering a rule and discovering another after learning the first.Source: §6 · p. 24
- The paper conjectures that conceptually or structurally different rules will not be near each other, and that more complex rules are harder to learn, though no well-defined difficulty metric is established.The hypothetical difficulty space could be quasi-metric, with d(A,B) not necessarily equal to d(B,A).Source: §6 · pp. 24–25
- Transferability is framed as discovering a second rule after succeeding on a first; these experiments continued learning rather than freezing internal parameters.The authors suggest that learning the first rule may prepare a learner for the second, but that preparation need not be symmetric or positive.Source: §6 · pp. 24–25
- The authors conjecture that learning two component rules as concepts makes their combined rule easier to discover.This is presented as a conjecture, not an established result.Source: §6 · p. 25
- The authors suggest that a digital-twin learner might select teaching schedules if human and machine performance could be situated in a shared difficulty space.The passage characterizes this prospect as close to science fiction.Source: §6 · p. 25
- The paper proposes Cognodynamics as a possible new science, with joint computer-science and psychology laboratories developing instruments and metrics.The proposed name is by analogy with the science of heat engines.Source: §6 · p. 25
A Additional Experiments
The appendix materials describe additional rule, difficulty-ordering, and transfer experiments, but the supplied passages do not report their outcome values.
A Additional Experiments
- Tables 5 and 6 list rules used in the rule-based experiments and their descriptions.Source: Table 5 · p. 27, Table 6 · p. 27
- Tables 7–10 present difficulty orderings for FC and OC rules or properties, including a majority ranking across three metrics for the rule tables.The listed rule-table captions state that lower metric values denote better performance.Source: Table 7 · p. 29, Table 8 · p. 30, Table 9 · p. 31
- The metric medians are based on five independent runs, and the range is described as an approximately 94% confidence interval for the median estimate.Source: A Additional Experiments · pp. 31–32
- Tables 11–12 and Figure 11 cover sequential and partial transfer experiments, including compound rules under different transfer curricula in FC and OC models.Figure 11's panels show performance metrics for compound rules; the supplied captions do not state comparative outcomes.Source: Table 11 · p. 33, Table 12 · p. 34, A Additional Experiments · p. 35