Source-linked AI summary
AI Learning and Conceptual Transfer in the Game of Hidden Rules
Christo Mathew, Wentian Wang, Jacob Feldman, Lazaros K. Gallos, Paul B. Kantor, Vladimir Menkov, Hao Wang
TL;DR
The paper addresses learning hidden rules from sparse feedback and examines whether learned structure transfers across related tasks and reflects human gameplay patterns. It uses reinforcement-learning agents and behavioral sequence analysis to study transfer, representation, and assisted versus unassisted play. Results identify structural alignment as central to compound-rule transfer and show that temporal outcome patterns can distinguish gameplay conditions, while the exploratory analyses remain limited by run counts and incomplete transfer coverage.
Problem
The work studies latent-rule learning from sparse accept/reject feedback in GOHR and asks how learned structure transfers across rules and relates to human gameplay behavior.
Method
The study combines Transformer-based Actor-Critic reinforcement-learning experiments with behavioral sequence analysis of pseudo-bot-assisted and unassisted two-player games.
Results
Structural alignment primarily governs compound-rule transfer, while temporal patterns of successes and failures distinguish assisted from unassisted gameplay behavior.
Takeaways & Limitations
Object-centric representations support stronger compositional reuse of relational abstractions but are more sensitive to curriculum mismatch and ordering effects.
Takeaways & Limitations
The conclusions are exploratory because experiments used relatively few independent runs and many comparisons, while MDS geometry required imputation for missing transfer combinations.
Abstract
from arXiv · showhide
This report summarizes the work conducted on the Game of Hidden Rules (GOHR), focusing on reinforcement learning agents trained to infer hidden rules from trial-and-error feedback, representation design, rule difficulty analysis, transfer learning, generalization, and pseudo-bot-assisted human learning analysis. The report focuses on the Transformer-based A2C framework, Feature-Centric and Object-Centric representations, experimental findings, and classification of human learning data.
1 Introduction
The work studies how agents infer latent rules in GOHR from sparse accept/reject feedback, with emphasis on representation, difficulty, transfer, generalization, and human-learning comparisons. It develops and evaluates Transformer-based Actor-Critic agents using Feature-Centric and Object-Centric representations across these analyses.
- Motivation and problem: GOHR examines latent-rule learning when agents receive only sparse feedback about whether actions satisfy hidden constraints.
- Research questions: The study asks how state representation affects learning and generalization, which rule properties drive difficulty, and whether learned knowledge transfers across related rules.
- Research questions: The study also examines whether reinforcement-learning agents exhibit behavior patterns similar to human learners.
- Representations and framework: Feature-Centric representations encode spatial feature maps, whereas Object-Centric representations explicitly encode individual objects and attributes.
- Representations and framework: Both representations are trained with a Transformer-based Actor-Critic framework and evaluated on independent learning, generalization, transfer learning, and transfer geometry.
- Scope and contributions: The report analyzes rule difficulty, related shape and color generalization, simple and compound transfer, transfer geometry, and pseudo-bot-assisted human gameplay.
2 The Game of Hidden Rules
GOHR is a 6 × 6 board task in which players infer hidden object-to-bucket rules through interaction. The environment returns move-status and episode-status feedback while preserving history-augmented observations in the reported experiments.
- Board and objects: GOHR uses a 6 × 6 grid with four corner buckets and randomly places n = 9 objects, each characterized by shape and color.
- Board and objects: Rules map object properties such as color, shape, and position to valid bucket assignments.
- Actions and rules: At each timestep, the player selects an object and assigns it to a bucket, while rule validity may depend on spatial, sequential, or conditional constraints.
- Feedback and termination: The episode continues while movable objects remain, finishes when all objects are removed, and reaches stalemate when objects remain but none can move.
- Feedback and termination: The environment reports ACCEPT for valid moves, DENY for invalid moves, and IMMOVABLE when the selected object cannot be moved.
- Observation pipeline: Reported experiments use legacy observations containing the current board, previous successful states, and their corresponding actions rather than only the current state.
3 Methodology
GOHR is formulated as partially observable reinforcement learning: agents infer hidden rules from accept/reject feedback using history-augmented FC or OC representations and a Transformer-based A2C model. FC separates attributes across spatial channels, whereas OC binds each piece’s attributes into object vectors before policy and value prediction.
- Reinforcement Learning Formulation: The agent observes encoded board states, selects object-to-bucket actions, and receives feedback while learning a policy and critic for hidden-rule inference.The objective is to minimize incorrect moves and place all pieces into correct buckets.
- Feature-Centric Representation: FC encodes four shapes and four colors as separate binary feature maps over the 6 × 6 board, requiring spatial alignment to associate attributes with pieces.Each piece activates one shape channel and one color channel at its position.
- Object-Centric Representation: OC represents each game piece as a 20-dimensional object vector containing one-hot color, shape, and x- and y-position features.Fixed object slots preserve associations among an object’s attributes and reduce the need to infer those bindings.
- Temporal State Construction: GOHR inputs include the current board plus the six most recent successful states and corresponding actions, creating a temporal context of seven.This history lets the model compare accepted moves and infer consistent feature-bucket or object-bucket relationships.
- Transformer-Based A2C: The Transformer projects encoded states into embeddings, adds positional and temporal information, and uses self-attention before actor and critic outputs.Self-attention captures dependencies among features, objects, positions, and previous actions.
- Transformer-Based A2C: The actor produces masked action logits for valid choices, while the critic supplies value estimates used to compute advantages for A2C training.Action masking excludes empty positions or unavailable objects.
4 Environmental and Training Setup
Experiments use the 6 × 6 GOHR environment with nine objects, four buckets, and hidden rules spanning feature, spatial, ordering, and relational structures. FC and OC agents share the Transformer-A2C architecture, while convergence is assessed using move-level and episode-level stability criteria.
- Environment: Experiments use a 6 × 6 board with nine objects and four buckets, where each episode begins with an unobserved rule defining correct bucket assignments.Agents infer the rule through interaction rather than observing it directly.
- Rule Suite: The evaluation suite spans feature-based, spatial, ordering-based, and relational hidden rules with varied difficulty and structural complexity.The complete rule list is provided in Appendix B.
- Model Comparison: FC and OC agents use the same Transformer-based A2C architecture but differ in state representation and action-space formulation.This isolates representation and action-space differences between the two models.
- Model Configuration: The training configuration is shared across reinforcement-learning experiments and summarized in Table 2.The supplied setup identifies Table 2 as the source of the common hyperparameters.
- Model Configuration: γ = 0.001 places most learning signal on immediate rewards because GOHR rewards are sparse and negative-only.The objective is minimizing incorrect moves rather than optimizing long-horizon cumulative reward.
- Convergence Evaluation: Convergence requires at least 10 consecutive successful moves plus jointly satisfied mean-window and max-window error criteria over a five-episode sliding window.The combined criteria are intended to prevent premature convergence and test sustained low-error behavior.
5 Experiments and Results
Rule difficulty depends strongly on the abstraction and structure required by each property, with feature-centric and object-centric representations producing different difficulty patterns. Object-centric learning is more uniformly difficult across properties and yields more stable cross-metric rankings, while feature-centric performance varies more with the convergence metric and generalizes poorly to shifted positions.
- Difficulty is largely determined by the abstraction level of the feature underlying each rule property.
- FC Model: In the FC model, positional properties are easiest, followed by feature-based properties, while feature_ordering and conditional properties are most challenging.Quadrant_mapping, proximity, and reading_order are easiest; feature_to_bucket and all_pieces_of_feature are moderately difficult, whereas bucket_ordering, feature_ordering, and conditional properties are harder.
- OC Model: In the OC model, property difficulty differences are smaller, with direct feature mappings and quadrant_mapping easiest, reading_order and proximity slower, and feature_ordering plus conditional properties hardest.Bucket_ordering is slightly harder than the easiest properties, while feature_ordering and conditional properties retain the slowest learning curves.
- Metric stability: OC rule difficulties cluster more closely than FC difficulties, making OC ordering harder from final metrics but preserving a more coherent relative ordering across metrics.FC rankings show crossings, especially among feature-dependent properties, whereas OC shows substantially fewer crossings and more stable rankings.
- Rule-level difficulty: Feature_ordering rules are highly challenging, and adding conditional properties makes colOrdL1_BRKY and shaOrdL1_qcts the most difficult rules.
6 Human Learning Analysis via Pseudo-Bot Modeling
The analysis classifies pseudo-bot-assisted versus unassisted human gameplay from filtered temporal outcome sequences rather than board states or explicit strategies. An interpretable Markov/backoff model achieved above-chance discrimination across assistance levels, with performance varying substantially by condition.
- Objective: The analysis classifies whether gameplay sequences are more consistent with pseudo-bot-assisted or unassisted human behavior.It uses two-player GOHR games with varying pseudo-bot assistance levels.
- Preprocessing: Gameplay transcripts are grouped by player, rule, and mover, chronologically sorted, and converted into filtered accepted-or-denied outcome sequences.Intermediate states and boundary or reset events are excluded from the resulting sequence.
- Model: The classifier models the next outcome using the previous valid outcome, a discretized cumulative success-rate context, and behavioral class.Separate class models use Laplace smoothing and interpolated lower-order backoff probabilities to reduce sparsity.
- Results: The Markov/backoff classifier achieved above-chance discrimination across all evaluated assistance levels, although performance varied substantially across conditions.Evaluation used accuracy, precision, recall, F1-score, and specificity under test-only and combined train+test settings.
- Results: 93.5% accuracy and 0.964 F1-score were achieved for ph4 under test-only evaluation, the strongest reported configuration.The ph4 setting is described as the fastest-learning pseudo-bot configuration.
- Results: Across most configurations, F1-scores ranged approximately between 0.55 and 0.86, indicating meaningful information in temporal outcome patterns.Some rules appeared more difficult than others for distinguishing assistance from non-assistance.
- Limitations: The model is interpretable and simple, but it ignores richer behavioral information and the effect of sequence length was not evaluated systematically.Limitations include binary outcomes, discretized success rates, and omission of reaction times, object identities, spatial configurations, and longer-term dependencies.
7 Overall Discussion and Conclusion
The discussion links hidden-rule learning outcomes to representation, rule structure, and structural transfer. Object-Centric representations were more stable and interpretable, while the exploratory conclusions remain bounded by limited runs, incomplete rule coverage, imputed transfer data, and simple agent mechanisms.
- Representation: Object-Centric representations consistently produced more stable and interpretable behavior than Feature-Centric representations across learning, transfer, clustering, and transfer-geometry analyses.OC systems generally learned rules more efficiently and showed stronger positive transfer within related rule families.
- Rule difficulty: Feature-mapping rules were generally easier to learn, whereas ordering-based, spatial, and compound rules produced greater difficulty, slower convergence, and higher variability.The analysis relates difficulty to increasing relational abstraction and compositional complexity.
- Transfer learning: Positive transfer was strongest when predecessor and target rules shared meaningful relational components, while unrelated or partially related curricula could produce weaker transfer or interference.Full-component pretraining consistently accelerated compound-rule learning.
- Transfer geometry: OC developed a comparatively coherent transfer geometry, with structurally related rules repeatedly forming stable clusters across clustering and MDS analyses.FC showed weaker alignment between predecessor and successor organization.
- Human learning: Temporal success-and-failure patterns contained sufficient statistical structure to distinguish assisted from non-assisted gameplay without explicit board-state information.The Markov/backoff model used short-term outcome transitions and cumulative success-rate context.
- Conclusion: The findings suggest that object-level representations better support relational abstraction, transferable learning, and coherent transfer organization in hidden-rule environments.The conclusion attributes hidden-rule learning behavior to both task structure and the representation used to encode it.
- Limitations: The conclusions are exploratory because many experiments used few independent runs and hundreds of comparisons, limiting statistical robustness and confidence.The authors state that additional runs would improve robustness and that the work helps define a path toward principled exploration.
- Limitations: The study covers only a subset of GOHR’s hidden-rule space, and missing transfer combinations required imputation that may influence inferred geometry.More complex rule families and complete transfer coverage could reveal additional behaviors.
A.4 Legacy Compatibility Wrappers
Legacy-compatible wrappers reconstruct the original history-augmented observations within the Gymnasium framework while retaining a standard current-state mode. This allows the same environment to operate in either mode.
- Wrapper design: The wrappers maintain previous successful states and previously executed actions to reconstruct legacy observations.The reconstructed observations match those used in earlier GOHR studies.
- Compatibility: The same environment can operate in standard Gymnasium mode or legacy-compatible mode.The legacy-compatible mode preserves compatibility with previous GOHR experiments.
A.5 Observations from the Refactoring Experiment
The refactoring experiment compared current-state Gymnasium observations with legacy history-augmented observations. Removing history degraded learning for history-dependent rules, motivating compatibility wrappers and alternative memory mechanisms.
- Observation designs: The default Gymnasium design exposes only the current board state, whereas the legacy environment includes previous successful states and actions.Temporal history is not automatically included in Gymnasium observations.
- Clockwise-rule comparison: For the clockwise rule, both Feature-Centric and Object-Centric agents learned successfully with legacy observations, but Feature-Centric learning failed with current-state observations and Object-Centric dynamics changed markedly.Figure 18 compares the two observation pipelines using dark-background legacy=True and white-background legacy=False TensorBoard plots.
- Learning effects: Several history-dependent rules, including clockwise and counter-clockwise ordering rules, showed substantially degraded learning when history information was removed.Similar behavior was observed for other history-dependent rules.
- Interpretation: The legacy observation pipeline contains information important for solving certain GOHR rule families, whereas the simplified observation design was not an adequate replacement.The Gymnasium refactoring reproduced game mechanics and rule definitions but not the relevant observation information.
- Implications: Future alternatives include recurrent architectures, explicit memory modules, or history wrappers that preserve standard reinforcement-learning interfaces.The results do not establish that Gymnasium-based implementations are unsuitable for GOHR generally.
A.6 Rendering and Visualization
The refactored GOHR environment adds rendering and visualization support for inspecting gameplay, debugging, and agent behavior. It also provides a publicly available foundation for future GOHR studies.
- Figure 19 displays the current board state, bucket contents, and feedback from the most recent action.
- The rendering interface supports interactive inspection of gameplay and visualization of agent behavior during training and evaluation.The environment can be accessed through env.render(mode="human").
- The source code for the Gymnasium-compatible GOHR environment is publicly available on GitHub.
- The implementation provides a foundation for reinforcement learning, transfer learning, curriculum learning, and human-agent comparison studies.
- The rule descriptions use bucket and quadrant indices from 0 to 3, with reading order defined left-to-right and top-to-bottom.
Rules Used in Rule-Based Experiments
The rule-based experiments use a catalog of explicitly described rules, while FC and OC transformer rules are organized by increasing difficulty based on performance measures. Transfer ratios distinguish positive from negative transfer.
- Table 6 lists the rules used in the Rule-Based experiments.
- FC and OC rule properties are arranged in increasing difficulty using M⋆ with a window size of 15.
- FC and OC transformer rule categories are ordered from easier to harder based on overall performance across M⋆ and E⋆.
- Reported rule-category values are medians across five runs, and category membership is not intended to impose a strict ordering.
- Transfer ratios below 1 indicate positive transfer, whereas values above 1 indicate negative transfer.
Clustering and MDS Results
The appendix applies a common analysis pipeline to FC and OC transfer matrices and extends the main-text analysis with clustering, MDS, and tanglegram visualizations.
- The FC and OC figures use the same analysis pipeline, changing only the input file from FC_diff_matrix.csv to OC_diff_matrix.csv.
- The appendix provides additional hierarchical clustering, multidimensional scaling, and tanglegram visualizations of FC and OC transfer geometry.
G.1 Hierarchical Clustering Results
Hierarchical clustering analyzes FC and OC transfer matrices by clustering rows and columns after separate mean imputation procedures. The resulting clusters characterize predecessor and successor transfer behavior, with OC showing stronger separation among rule families.
- Clustering procedure: The clustering analysis reads transfer matrices from FC_diff_matrix.csv and labels rows and columns for visualization.
- Clustering procedure: Row clustering uses column-mean imputation, Euclidean distances, and hierarchical clustering of the resulting matrix.
- Clustering procedure: Column clustering uses row-mean imputation, transposed-matrix distances, and hierarchical clustering.
- FC results: FC successor-rule clusters group rules with similar sensitivity to predecessor-rule experience.
- FC results: FC predecessor-rule clusters group rules producing similar transfer effects on successor-rule learning.
- OC results: OC predecessor-rule clusters separate ordering-based, feature-based, and spatial rule families more strongly than corresponding FC clusters.
G.2 Multidimensional Scaling Results
The section visualizes rule-transfer and influence structures with three-dimensional MDS embeddings and tanglegrams, comparing Feature-Centric (FC) and Object-Centric (OC) representations. The visualizations show representation-dependent clustering, with OC producing more compact and separable structurally related successor-rule clusters.
- MDS embeddings: 3-dimensional MDS embeddings represent FC successor-rule transfer behavior and predecessor-rule influence, with distances encoding behavioral similarity.The visualizations distinguish how predecessor experience affects successor learning from how predecessor rules influence successor learning.
- MDS embeddings: OC successor-rule embeddings form more compact and separable clusters for structurally related rules than FC embeddings.This indicates clearer structural organization in the OC visualization.
- Tanglegram comparisons: FC row-versus-column tanglegrams show weaker agreement between predecessor-rule influence and successor-rule sensitivity.The large number of crossing connections visualizes this weaker alignment.
- Tanglegram comparisons: FC and OC predecessor-rule clustering structures align relatively weakly, while some successor-rule relationships remain preserved across representations, especially for spatial and ordering-based rules.The comparisons use pruned dendrograms of common rules and report cophenetic correlation, Baker’s gamma, and entanglement.