PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Learning and Conceptual Transfer in the Game of Hidden Rules

Christo Mathew, Wentian Wang, Jacob Feldman, Lazaros K. Gallos, Paul B. Kantor, Vladimir Menkov, Hao Wang

arXiv:2608.21372v1cs.AIcs.LG

TL;DR

Learning latent rules from sparse accept/reject feedback is important but underexplored. This report evaluates reinforcement-learning agents and behavioral sequence models in GOHR, finding that transfer depends on structural alignment and temporal behavior patterns distinguish assisted from unassisted play.

  • Problem

    The work addresses limited evidence on learning latent rules from sparse accept/reject feedback in the GOHR environment.

    Source: §1.2 · p. 4

  • Method

    The report studies Transformer-based A2C agents with feature- and object-centric representations, transfer and generalization experiments, and probabilistic modeling of human gameplay.

    Source: §2.1 · p. 6, §2.2.1 · p. 6

  • Results

    Transfer depends primarily on structural alignment, while simple probabilistic sequence models distinguish assisted from unassisted gameplay behavior in this dataset.

    Source: §5.4.5 · p. 37, §6.6 · pp. 48–49

  • Takeaways & Limitations

    Object-centric representations support compositional reuse but are sensitive to curriculum mismatch and ordering effects, and temporal behavior patterns provide interpretable evidence of assistance.

    Source: §5.4.5 · p. 37, §6.6 · p. 48

    The exploratory conclusions are limited by relatively few independent runs and hundreds of comparisons, preventing claims of established statistical confidence.

    Source: §7.2 · p. 50

Abstract

from arXiv · show

This report summarizes the work conducted on the Game of Hidden Rules (GOHR), focusing on reinforcement learning agents trained to infer hidden rules from trial-and-error feedback, representation design, rule difficulty analysis, transfer learning, generalization, and pseudo-bot-assisted human learning analysis. The report focuses on the Transformer-based A2C framework, Feature-Centric and Object-Centric representations, experimental findings, and classification of human learning data.

1 Introduction

This work uses the Game of Hidden Rules (GOHR) to study how agents infer latent rules from sparse accept/reject feedback, focusing on representation, difficulty, generalization, transfer, and human-learning comparisons. It develops and evaluates Feature-Centric and Object-Centric representations within a transformer-based Actor-Critic framework.

1.1 Motivation

  • GOHR tests whether agents can infer an unknown rule governing object placement using only sparse feedback indicating whether actions are accepted or rejected.The setting supports controlled study of abstraction, compositionality, rule difficulty, generalization, transfer, and comparisons between artificial and human learning.Source: §1.1 · p. 4

1.2 Problem Statement

  • The study compares Feature-Centric representations of spatial feature maps with Object-Centric representations of explicit object attributes, trained using a transformer-based Actor-Critic framework.The resulting agents are examined through independent rule learning, generalization, transfer learning, and transfer-geometry analyses.Source: §1.2 · p. 5

1.3 Summary of Contributions

  • The work develops and compares Object-Centric and Feature-Centric representations (Pulick, Bharti, et al. 2022), introduces a transformer-based Actor-Critic architecture (Mathew et al. 2025), and analyzes rule difficulty, related-rule generalization, simple-to-simple and compound-rule transfer, and transfer geometry.Source: §1.3 · p. 5

2 The Game of Hidden Rules

GOHR is a controlled 6 × 6 grid-world for discovering hidden object-to-bucket rules through trial-and-error. Its experiments use a history-augmented legacy interface that differs from the later Gymnasium refactoring.

2.1 Overview of GOHR

  • GOHR places nine randomly initialized, shape-and-color objects on a 6 × 6 board with four corner buckets governed by a hidden rule.Players or agents must infer the rule and place every object into a valid bucket.Source: §2.1 · p. 6

2.2.1 Board Layout

  • The board uses grid indices 1–36, while spatial coordinates (x, y) specify each cell’s true layout.The four buckets occupy the board’s corners.Source: §2.2.1 · p. 6

2.2.2 Pieces, Buckets, and Rules

  • Each action selects an object and assigns it to a bucket, while rules may depend on spatial properties, sequential constraints, or attribute dependencies.Objects are characterized by shape, color, and position, and assignment validity depends on the hidden rule.Source: §2.2.2 · pp. 6–7

2.2.3 Feedback and Episode Termination

  • Feedback marks moves as ACCEPT, DENY, or IMMOVABLE, while episodes CONTINUE when movable objects remain, FINISH when all objects have been removed, or reach STALEMATE when remaining objects cannot be moved.The corresponding move codes are 0 for ACCEPT, 4 for DENY, and 7 for IMMOVABLE; episode outcomes are coded 0, 1, and 2, respectively.Source: §2.2.3 · p. 7

2.3 GOHR Environment

  • All reported experiments use the legacy GOHR interface, whose observations include the current board plus previous successful states and actions rather than only the current state.A Gymnasium refactoring was developed, but preliminary current-state-only observations did not reproduce the legacy learning behavior.Source: §2.3 · p. 8

3 Methodology

GOHR is formulated as a partially observable reinforcement-learning task in which agents infer hidden rules from encoded board states, actions, and rewards. The methodology compares Feature-Centric and Object-Centric representations within a Transformer-based A2C agent that uses temporal history, masked actions, sparse rewards, and complementary convergence metrics.

3.1 Reinforcement Learning Formulation

  • The agent learns a policy and value function from encoded states and reward feedback to minimize incorrect moves while placing pieces into rule-defined buckets.The hidden rule is unobserved, so the task requires inference from interaction with the environment.Source: §3.1 · p. 8

3.2.1 Feature-Centric Representation

  • Feature-Centric encoding uses eight binary spatial channels for four shapes and four colors, requiring the model to infer object associations through aligned feature maps.A piece activates one shape channel and one color channel at its grid position rather than a fused object representation.Source: §3.2.1 · pp. 9–10, Figure 2 · p. 10

3.2.2 Object-Centric Representation

  • Object-Centric encoding binds each piece’s one-hot color, shape, and position into a 20-dimensional vector, preserving attribute associations in fixed object slots.The implementation can append four bucket dimensions to historical object representations, producing 9 × 24 tensors.Source: §3.2.2 · pp. 10–11

3.2.3 Temporal History Encoding

  • Temporal inputs combine the current board with six recent successful states and actions, enabling inference of consistent feature-bucket or object-bucket relationships.The resulting temporal context length is seven.Source: §3.2.3 · pp. 11–12

3.3 Action Space

  • Actions assign a board position or object to one of four buckets, with invalid selections excluded by an action mask.The FC action space is position-based, whereas the OC action space is object-based.Source: §3.3 · p. 12

3.5 Transformer-Based Actor-Critic Architecture

  • The Transformer processes projected tokens with positional and temporal embeddings and self-attention, producing shared representations for masked actor logits and critic value estimates.Self-attention captures dependencies among features, objects, positions, and previous actions relevant to hidden-rule inference.Source: §3.5.2 · pp. 13–14, §3.5.1 · p. 13, §3.5.3 · p. 14

3.6 A2C Learning Algorithm

  • A2C combines discounted returns, advantage-based actor-critic optimization, entropy regularization, and ε-greedy exploration to train the policy and value networks.Entropy regularization encourages exploration and prevents premature determinism, while ε-greedy sampling selects random valid actions with probability ε_t.Source: §3.6 · pp. 14–15

3.7 Evaluation Metrics

  • Evaluation uses move-level, mean-window, and maximum-window convergence metrics that measure stability, average convergence, and worst-case robustness across training runs.The move-based metric is aggregated as the median m⋆ across runs, and correlations are examined across 18 rules with 6-step and 8-step memory windows.Source: §3.7 · pp. 15–16

4 Environmental and Training Setup

Experiments used the GOHR 6×6 environment to test agents inferring hidden rules through interaction. Feature-Centric and Object-Centric Transformer-based A2C agents were trained and evaluated under defined convergence criteria.

4.1 Environment and Rule Suite

  • Experiments used GOHR’s 6×6 environment with nine objects and four buckets, testing hidden rules spanning feature-based, spatial, ordering-based, and relational structures.The complete rule list and descriptions are provided in Appendix B.Source: §4.1 · p. 16

4.2 Model Configuration and Hyperparameters

  • Feature-Centric and Object-Centric agents shared the Transformer-based A2C architecture but differed in state representation and action-space formulation.Training hyperparameters are summarized in Table 2; the discount factor was γ = 0.001 because rewards are sparse and negative-only, emphasizing immediate error minimization.Source: §4.2 · p. 17, Table 2 · p. 17

4.3 Training and Evaluation Protocol

  • Each experiment used five random-seed runs trained to convergence, jointly requiring ten consecutive successful moves and stable mean- and max-window error rates over five episodes.The combined criteria prevented premature convergence by assessing both sustained correct-action sequences and stable low-error behavior over time.Source: §4.3 · pp. 17–18

5 Experiments and Results

This section evaluates rule learning, difficulty, metric stability, representation-specific generalization, and transfer in the Game of Hidden Rules. Across these analyses, it examines how Feature-Centric and Object-Centric representations support relational, compositional, and structurally coherent transfer under varying scene complexity and rule alignment.

5 Experiments and Results

  • Experiments used rule-file and trial-list configurations, with the latter specifying rules and initial board generation while supporting transfer experiments.The study analyzed eighteen rule files, training each rule independently to assess isolated rule learning.Source: §5 · p. 18, §5.1 · p. 18

5.1.1 Difficulty Analysis of Rule Properties

  • Feature-Centric learning ranked positional properties easiest, followed by feature mappings, bucket ordering, and finally feature ordering and conditional properties.Quadrant_mapping was easiest, while feature_ordering and conditional properties had very slow learning curves.Source: §5.1.1 · p. 20
  • Object-Centric property difficulties were more clustered: feature mappings and all_pieces_of_feature were easiest, bucket_ordering was slightly harder, and feature_ordering and conditional properties remained hardest.Reading_order and proximity were also slower to learn because positional information had to be derived from x and y coordinates.Source: §5.1.1 · p. 21

5.1.2 Rule Difficulty Analysis

  • Complete-rule difficulty showed stable category-level ordering across all three metrics, although ordering within categories remained somewhat ambiguous.This analysis grouped eighteen independently trained rule files by conceptual properties and metric ranges.Source: §5.1 · p. 18, §5.1.2 · p. 21

5.1.3 Key Observations

  • FC rule rankings varied across convergence metrics, whereas OC produced fewer crossings and a more coherent ordering, with feature mappings easier and feature ordering or conditional rules hardest.Episode-level error-rate rankings and move-level correct-action rankings disagreed most within related feature-dependent categories.Source: §5.1.3 · p. 22

5.2.1 Feature-Centric (FC) Generalization

  • Feature-Centric OOD errors commonly reached 55–75% versus 25–35% ID, with quadMixed1, quadNearby, ordL1, and ordL1_Nearby exceeding 80% OOD error under unseen positions.The checkerboard split restricted training to white-square positions and tested placements across the full board, as shown in Fig. 7 and Fig. 8.Source: §5.2.1 · p. 24, Figure 7 · p. 24, Figure 8 · p. 25

5.2.2 Object-Centric (OC) Generalization

  • Object-Centric error increased when evaluation expanded from 9 to 16 objects, but local color- and shape-to-bucket rules degraded less than ordering and large-set relational rules.The model was trained only on 9-object boards and evaluated on both 9-object ID and 16-object OOD scenes, as shown in Fig. 9.Source: §5.2.2 · p. 25, Figure 9 · p. 26
  • Object-Centric representations transferred learned rule structure to denser scenes, although performance remained sensitive to increases in ordering, relational, and global spatial complexity.The results indicate meaningful out-of-distribution generalization despite degradation as scene cardinality increases.Source: §5.2.2 · p. 26

5.2.3 Observations

  • Increasing scene cardinality from 9 to 16 generally reduces OC performance, with ordering and relational rules especially sensitive to larger object sets.For FC, reading-order and spatially structured rules such as ordL1, ordL1_-, Nearby, quadNearby, and quadMixed1 show strong degradation under distribution shifts.Source: §5.2.3 · pp. 26–27

5.3.3 Effect of Increasing the Number of Runs

  • Increasing runs from 5 to 15 raised statistically indistinguishable comparisons from 40/60 (66%) to 50/60 (83%), improving conclusion stability.The larger pooled analysis serves as a robustness check, while the original five-run analysis still captures the overall similarity trends.Source: §5.3.3 · p. 28, §5.3.4 · p. 29

5.3.1 Within-Rule Comparison

  • Repeated runs of each rule generally produce consistent learning behavior, with at least one statistically indistinguishable metric for every rule.The within-rule analysis applied Kruskal–Wallis tests across five runs per metric and found that variability remained for some metrics.Source: §5.3.1 · p. 27, §5.3.4 · p. 28

5.3.2 Comparison Between Shape and Color Rules

  • Across 30 shape–color rule comparisons, 29 showed no statistically significant difference, supporting highly similar performance characteristics between corresponding rules.The exception was (col1Ord_BRKY, sha1Ord_qcts) under FC using the m⋆ metric, while its other two metrics were non-significant.Source: §5.3.4 · p. 28, §5.3.2 · p. 28

5.4.1 Simple-to-Simple Transfer

  • FC transfer is positive within some related families but can severely degrade across incompatible rules, reflecting unstable reuse of learned positional or feature patterns.Bucket-ordering and feature-based pairs can transfer constructively, but cw is especially irregular, and quadrant-to-ordering or ordering-to-feature transfers can be strongly negative.Source: §5.4.1 · p. 31
  • OC transfers positively within bucket-ordering and feature-based families, with weaker mutual transfer for quadrant rules and partial compatibility for reading-order rules.Ordering-based and feature-based rules form the clearest coherent groups, while quadNearby and quadMixed1 show moderate mutual transfer and ordL1 and ordRevOfL1 occupy an intermediate position.Source: §5.4.1 · pp. 31–32
  • OC produces coherent family-level transfer, while FC transfer remains noisy, checkpoint-dependent, and dominated by positional heuristics.OC transfers positively within related rule families and shows systematic negative transfer between incompatible structures; FC exhibits inconsistent transfer, including occasional negative self-transfer.Source: §5.4.1 · pp. 31, 33

5.4.2 Compound Transfer Experiments

  • Compound-transfer experiments compare sequential, partial/mixed, and unrelated transfer protocols before training on a compound rule formed by combining two component rules.For all transfer phases, ϵ was reinitialized to ϵstart at the beginning of each stage to encourage exploration and avoid premature convergence to inherited policies.Source: §5.4.2 · p. 33

5.4.3 Transfer to ordL1_Nearby

  • OC pretraining on ordL1 and quadNearby components reduces normalized convergence to approximately 0.2, an almost five-fold improvement over baseline, whereas FC gains remain modest.OC can also benefit from partially compatible unrelated curricula, but incompatible curricula produce stronger negative transfer than in FC.Source: §5.4.3 · p. 35

5.4.4 Transfer to cm_RBKY_cw_0123

  • For cm_RBKY_cw_0123, full-component pretraining lowers OC’s normalized convergence ratio to approximately 0.5, while cm_RBKY alone transfers better than cw_0123 alone.In FC, cm_RBKY alone yields approximately 0.13; in OC, cw_0123 alone produces ratios exceeding 1.7, and incompatible curricula can exceed 3.Source: §5.4.4 · p. 36

5.4.5 Summary

  • Full-component pretraining produces the strongest transfer, while unrelated curricula generally cause negative transfer; OC shows larger gains and degradations than FC.Transfer is strongly order-dependent, and single-component benefits depend on the component’s structural role in the target rule.Source: §5.4.5 · pp. 36–37

5.5 Multidimensional Scaling Analysis

  • MDS based on Mann–Whitney-derived dissimilarities finds several rule clusters, while the third dimension resolves a large cluster that overlaps in the first two dimensions.The embedding treats similarity as approximately equal learning difficulty and leaves open whether clusters reflect algorithmic and conceptual rule structure.Source: §5.5 · p. 38, Figure 13 · p. 38, Figure 14 · p. 39

5.6 Transfer Geometry and Clustering Structure

  • Transfer geometry compares rules by their predecessor–successor transfer patterns, using transfer differences relative to baseline and mean imputation for missing entries.Positive differences indicate reduced convergence time, whereas negative differences indicate interference; row and column analyses capture successor sensitivity and predecessor influence, respectively.Source: §5.6 · pp. 39–40

5.6.1 Row clustering: similarity of successor-rule behavior.

  • OC row clustering groups spatial, feature-based, and bucket-ordering rules coherently, whereas FC clusters are weaker, less separated, and include unstable outliers such as cw.The OC grouping is consistent with transferable feature-to-bucket abstractions and positive transfer within ordering-based families.Source: Figure 15 · p. 40, §5.6.1 · p. 41

5.6.2 Column clustering: similarity of predecessor-rule influence.

  • OC column clustering separates ordering, spatial, and feature-based predecessor influences more clearly than FC, whose embeddings overlap and place unrelated rules near one another.This weaker FC organization reinforces that FC reflects the conceptual structure of rules less effectively than OC.Source: Figure 16 · p. 41, §5.6.2 · p. 42

5.6.3 Comparison between row and column structures.

  • OC row and column structures agree comparatively strongly across major rule families, while FC shows weaker alignment and less unified predecessor–successor organization.Cross-representation comparisons find moderate similarity for rows but very weak similarity for columns; OC embeddings remain compact and separated, whereas FC geometry is distorted and unstable.Source: §5.6.3 · pp. 42–43
  • OC transfer geometry is more coherent and structured than FC, with related rules repeatedly clustering across dendrograms, MDS embeddings, and tanglegram comparisons.The organization remains comparatively stable between predecessor and successor views, whereas FC transfer behavior is more diffuse and internally inconsistent, suggesting stronger relational abstractions for systematic transfer across rules.Source: §5.6.3 · p. 43

6 Human Learning Analysis via Pseudo-Bot Modeling

This section shows that temporal outcome patterns can distinguish unassisted from pseudo-bot-assisted GOHR gameplay without board-state or handcrafted strategic features. An interpretable Markov/backoff model achieved above-chance performance, strongest for ph4, while results varied by assistance level and rule complexity.

6.1 Motivation; 6.2 Dataset and Problem Setup

  • The analysis classified whether two-player gameplay sequences were more consistent with bot-assisted or unassisted human behavior using temporal outcome patterns.Transcript moves were grouped by player, rule, and mover, then sorted chronologically so each sequence represented one player side interacting with a particular rule.Source: §6.2 · p. 44, §6.1 · p. 44

6.3.1 Outcome Sequence Representation

  • Raw gameplay was converted into A/D outcome sequences from object-count changes and turn transitions, excluding intermediate same-player states and boundary or reset events.Each timestep also included the previous valid outcome and a discretized cumulative success-rate context, with a dedicated start token and start bin.Source: §6.3.1 · pp. 44–45

6.3.2 Markov / Backoff Model

  • Separate class-conditional Markov/backoff models estimated next-outcome probabilities from prior outcomes and success-rate bins using Laplace smoothing and weighted interpolation.The interpolation placed most probability mass on the full contextual model while retaining lower-order backoff terms for sparse contexts.Source: §6.3.2 · p. 46

6.3.3 Sequence Classification

  • Classification compared sequence log-likelihoods under the two behavioral classes and predicted assistance when the resulting score exceeded zero.The classifier used the first K valid outcomes to assess whether temporal progression was more consistent with assisted or unassisted behavior.Source: §6.3.3 · p. 47

6.4 Evaluation Protocol

  • Evaluation covered ph4, ph8, ph12, and ph16 using accuracy, precision, recall, F1-score, and specificity in test-only and combined train+test settings.These settings tested classification across multiple pseudo-bot assistance levels and examined the stability of learned behavioral distributions.Source: §6.4 · p. 47

6.5.1 Bot-Assisted Side Classification

  • 93.5% accuracy and 0.964 F1-score were achieved for ph4 under test-only evaluation, while discrimination remained above chance across all evaluated assistance levels.Across most configurations, F1-scores ranged approximately between 0.55 and 0.86.Source: §6.5.1 · p. 47, Table 4 · p. 48

6.5.2 Effect of Number of Moves; 6.5.3 Rule-Level Differences

  • Sequence length and rule complexity qualify interpretation: length-performance relationships were not systematically evaluated, and complex spatial or relational rules may reduce separability.Simpler rules often produced more stable outcome dynamics, making assisted behavior easier to distinguish, but the complexity explanation was not directly tested.Source: §6.5.2 · p. 48, §6.5.3 · p. 48

6.6 Discussion

  • The approach captures behavioral differences through short-term transitions and cumulative performance context, but binary outcomes, success-rate discretization, and omitted reaction-time, object, spatial, and long-term strategic information limit sensitivity.Despite these limitations, the interpretable probabilistic model recovered meaningful distinctions between assisted and unassisted gameplay in this dataset.Source: §6.6 · pp. 48–49

7 Overall Discussion and Conclusion … A.3 Support for Multiple Representations

The work finds that object-level representations, structural alignment between rules, and rule complexity strongly shape hidden-rule learning, while limitations motivate broader experiments and improved curricula. Appendix A documents a Gymnasium-compatible refactoring that supports both representations but shows that history-augmented observations remain important for reproducing legacy performance.

7.1 Key Findings

  • Object-Centric representations produced more stable, interpretable, efficient, and transferable hidden-rule learning than Feature-Centric representations across the reported analyses.OC formed clearer transfer geometries, whereas FC was more diffuse and checkpoint-sensitive, including unstable transfer for the cw rule.Source: §7.1 · pp. 49–50
  • Rule difficulty increased with relational abstraction and compositional complexity: feature-mapping rules were easier, while ordering, spatial, and compound rules were harder.Compound rules combining multiple relational constraints were consistently among the most difficult tasks for both representations.Source: §7.1 · p. 49
  • Transfer was strongest when predecessor and successor rules shared meaningful relational components, with full-component pretraining accelerating learning and unrelated curricula sometimes causing interference.The findings support curricula that preserve structural continuity between tasks, particularly within ordering-based and feature-based families.Source: §7.1 · p. 49

7.2 Limitations

  • The conclusions remain exploratory because many experiments used few runs, hundreds of comparisons, an incomplete rule suite, imputed transfer matrices, simple agent mechanisms, and limited human behavioral features.The Gymnasium refactoring also lost board-and-move history and did not match legacy-system performance, while human analysis classified assistance rather than directly modeling reasoning.Source: §7.2 · pp. 50–51

7.3 Future Work

  • Future work should expand relational, temporal, and compositional rules; strengthen reinforcement learning; automate transfer-aware curricula; improve statistical coverage; and model richer human behavior.Proposed extensions include memory-augmented, offline, and model-based methods, complete transfer matrices, direct strategy modeling, and human-in-the-loop experiments.Source: §7.3 · p. 51

Gymnasium-Compatible GOHR Environment

  • The Gymnasium-compatible GOHR refactoring aimed to improve modularity, library integration, and separation of environment dynamics from state-representation construction.It was also intended to provide a conventional reinforcement-learning interface in which agents, rather than the environment, model temporal history.Source: Gymnasium-Compatible GOHR Environment · p. 52, §A.1 · p. 53

A.1 Design Goals

  • The refactoring pursued a standard Gymnasium interface, unified FC and OC access, and maintainable, extensible, debuggable environment design.Its API includes step(action) returning observation, reward, termination, truncation, and info values.Source: §A.1 · p. 53

A.2 Gymnasium Interface

  • The Gymnasium interface defaults to current-state observations without temporal history, unlike the legacy history-augmented pipeline, and preliminary experiments failed to reproduce legacy learning behavior for history-dependent rules.Consequently, the main-body experiments use the legacy observation pipeline, while the refactoring remains a software contribution and case study.Source: Gymnasium-Compatible GOHR Environment · p. 52, §A.2 · p. 53

A.4 Legacy Compatibility Wrappers

Optional wrappers preserve compatibility with previous GOHR experiments by reproducing the legacy observation pipeline. The environment can therefore operate in either standard Gymnasium or legacy-compatible mode.

A.4 Legacy Compatibility Wrappers

  • The same environment supports both standard Gymnasium operation and legacy-compatible operation through optional wrappers.Source: §A.4 · p. 53
  • Legacy-compatible observations maintain successful-state history, include previously executed actions, and construct history-augmented inputs matching earlier studies.Source: §A.4 · p. 53

A.5 Observations from the Refactoring Experiment

The refactoring experiment showed that replacing legacy history-augmented observations with Gymnasium’s current-state design substantially impaired learning of history-dependent GOHR rules. Legacy-compatible wrappers were consequently developed to preserve the original observations within Gymnasium.

Learning effects

  • For the clockwise rule, both FC and OC learned successfully with legacy observations, whereas current-state observations caused FC to fail and changed OC’s learning dynamics.Figure 18 compares the legacy=True and legacy=False observation pipelines.Source: §A.5 · p. 54, Figure 18 · p. 54
  • Removing observation history substantially degraded learning for clockwise, counter-clockwise, and several other history-dependent rules.The preliminary experiments found behavior markedly different from the legacy implementation when history information was removed.Source: §A.5 · p. 54

Interpretation and limitations

  • The legacy observation pipeline therefore contains information important for solving certain rule families, which the simplified Gymnasium representation did not adequately replace despite reproducing the game mechanics and rule definitions.The limitation concerns the observation design rather than Gymnasium implementations generally; recurrent architectures, explicit memory modules, or history wrappers are suggested alternatives.Source: §A.5 · pp. 54–55

Refactoring contribution

  • Legacy-compatible wrappers were developed to reconstruct the original history-augmented observations within Gymnasium and preserve compatibility with previous GOHR studies.The legacy environment directly incorporates previous successful states and actions, unlike Gymnasium’s default current-board-state observation.Source: §A.5 · pp. 54–55

A.6 Rendering and Visualization … Appendix C

The appendix documents visualization, public availability, and rule specifications for the Gymnasium-compatible GOHR environment. Together, these materials support debugging, future research, and interpretation of board rules.

A.6 Rendering and Visualization

  • The rendering interface enables interactive inspection of gameplay in the Gymnasium-compatible GOHR environment.It provides access to gameplay inspection for visualization and debugging.Source: §A.6 · p. 55
  • Visualization simplifies debugging and helps researchers examine agent behavior during training and evaluation.An example display includes the board state, bucket contents, and feedback from the most recent action.Source: §A.6 · p. 55, Figure 19 · p. 55

A.7 Availability

  • The environment source code is publicly available on GitHub.3Source: §A.7 · p. 56
  • The implementation provides a foundation for future reinforcement learning, transfer learning, curriculum learning, and human-agent comparison studies.Source: §A.7 · p. 56

Rule Descriptions

  • Appendix C catalogs rules governing shape assignment, color assignment, removal order, bucket targets, and clockwise filling.The listed rules include looped sequences, specified bucket mappings, reading-order removal, and quadrant-based selection.Source: Table 5 · p. 56, Rule Descriptions · p. 57
  • Rule descriptions use bucket and quadrant indices from 0 to 3, with top-left-to-bottom-right reading order and B0–B3 and Q0–Q3 notation.Source: Rule Descriptions · p. 57

Rules Used in Rule-Based Experiments … Appendix G

The appendices document the rules used in rule-based experiments, rank rule-property and overall difficulty for FC and OC transformer models, and provide a transfer heatmap for a compound rule. Difficulty rankings use M⋆ with window size 15, while overall categories summarize performance across M⋆ and E⋆.

Rules Used in Rule-Based Experiments

  • The rule-based experiments use the rule set listed in Table 6.Source: Table 6 · p. 58

Rule-Property Difficulty Rankings

  • FC rule properties are arranged by increasing difficulty according to M⋆, using a window size of 15, with highlighted properties marking combinations.Source: Table 7 · p. 59
  • OC rule properties are likewise arranged by increasing M⋆ difficulty with a 15-step window, and highlighted rows identify properties combined with others.Source: Table 8 · p. 60

Overall Rule Difficulty for FC and OC Models

Transfer Plots

  • The transfer plot reports normalized transfer ratios for the compound rule cm_RBKY_cw_0123, where values below 1 indicate positive transfer and values above 1 indicate negative transfer.Source: §5.4.2 · p. 34

Clustering and MDS Results

The section presents appendix visualizations for hierarchical clustering, multidimensional scaling (MDS), and tanglegram analyses of Feature-Centric (FC) and Object-Centric (OC) transfer geometry. FC and OC figures use the same analysis pipeline, differing only in the input matrix.

Clustering and MDS Results

  • FC and OC visualizations were generated with the same analysis pipeline using FC_diff_matrix.csv and OC_diff_matrix.csv, respectively.The input file was changed between representations while the analysis procedure remained constant.Source: Clustering and MDS Results · p. 64
  • The appendix provides hierarchical clustering, MDS, and tanglegram visualizations discussed in Section 5.6.Source: Clustering and MDS Results · p. 64
  • These figures show the complete set of FC and OC transfer-geometry results and complement the main-text analyses.Source: Clustering and MDS Results · p. 64

G.1 Hierarchical Clustering Results

Hierarchical clustering examined FC successor-rule transfer behavior, FC predecessor-rule influence, and OC predecessor-rule influence. The OC clustering showed stronger separation among ordering-based, feature-based, and spatial rule families than the corresponding FC clustering.

Clustering procedure

  • The hierarchical clustering procedure used distance-based clustering after column-mean imputation for row clustering and row-mean imputation for column clustering.The implementation computed distances and applied hierarchical clustering separately to rows and columns.Source: §G.1 · p. 64

FC successor-rule transfer behavior

  • FC successor-rule clusters contained rules with similar sensitivity to predecessor-rule experience.The analysis used hierarchical clustering to group rules by successor-rule transfer behavior.Source: Figure 21 · p. 65

FC predecessor-rule influence

  • FC predecessor-rule clusters contained rules producing similar transfer effects on successor-rule learning.This clustering characterized how predecessor-rule influence shaped successor-rule learning.Source: Figure 22 · p. 65

OC predecessor-rule influence

  • OC predecessor-rule influence clustered into more strongly separated ordering-based, feature-based, and spatial rule families than the corresponding FC clustering.Rules within each cluster exhibited similar transfer effects on successor-rule learning.Source: Figure 23 · p. 66

G.2 Multidimensional Scaling Results … Supplementary Materials

The supplementary analyses visualize transfer and influence structures through three-dimensional MDS embeddings and compare clustering agreement with tanglegrams across representations. Additional interactive visualizations and analysis outputs support the reported results.

G.2 Multidimensional Scaling Results

  • The MDS visualizations are implemented as rotatable three-dimensional row and column views with labeled rules and separate dimensions.The supplied code uses three-dimensional coordinates and produces row and column MDS visualizations.Source: §G.2 · p. 67
  • OC embeddings form more compact and separable clusters of structurally related rules than FC embeddings, while FC embeddings capture successor transfer and predecessor influence.The MDS views represent similarity in successor responses to predecessor experience and similarity in predecessor effects on successor learning.Source: Figure 24 · p. 68, Figure 25 · p. 68, Figure 26 · p. 69

G.3 Tanglegram Comparisons

  • Tanglegram comparisons align common rules, prune nonshared labels, and report cophenetic correlation, Baker’s gamma, and entanglement.The implementation compares clustering structures after retaining common rules.Source: §G.3 · p. 70
  • FC row and column clustering show weaker agreement, reflected by many crossing connections in their tanglegram.The comparison concerns predecessor-rule influence versus successor-rule sensitivity under the FC representation.Source: Figure 27 · p. 71
  • FC and OC organize predecessor-rule influence differently, as indicated by relatively weak alignment between their clustering structures.The tanglegram compares FC and OC predecessor-rule clustering structures.Source: Figure 28 · p. 72
  • Some high-level successor-rule relationships remain preserved across FC and OC, particularly for spatial and ordering-based rules.This cross-representation comparison is shown through successor-rule clustering structures.Source: Figure 29 · p. 73

Supplementary Materials

  • Supplementary materials provide additional interactive visualizations and analysis outputs supporting the results reported in the paper.The supplied passage identifies these materials as associated with the report.Source: Supplementary Materials · p. 74

Related papers

Loading 2608.21372v1…