PaperlayerView exact revision on arXiv

Source-linked AI summary

Groupoid-Based Internal State Representations for Reinforcement Learning with Local Symmetries

Ben Opperman, Eduardo Alonso, Esther Mondragón

arXiv:2609.13035v1cs.LGcs.AI

TL;DR

Standard RL can struggle to use experience efficiently and to generalize when environments contain local, context-dependent symmetries that global group actions do not capture. This paper uses groupoids, canonical representatives, and transporters to represent those local equivalences. It reports improved sample efficiency, convergence, and long-run stability over traditional methods in environments with repeated local structure.

  • Problem

    Global and uniform symmetry assumptions cannot capture many partial or context-dependent equivalences, while standard RL struggles with experience efficiency and generalization across repeated local structure.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    The framework uses canonical representatives and transporters to update values in canonical space and propagate learning along verified local morphisms.

    Source: §3.1 · p. 5, §3.5 · p. 6

  • Results

    In environments with repeated local structure, groupoid-based RL reduces effective sample complexity, accelerates convergence, and improves long-run stability over traditional tabular or symmetry-agnostic methods.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Groupoids provide a flexible way to represent partial symmetry, with potential relevance to robotics, networked systems, and structured simulation environments.

    Source: §5 · p. 8

    The study focuses on discrete, carefully controlled environments; broader realistic and stochastic testing and systems-level optimization remain future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Symmetries play a central role in reducing the complexity of reinforcement learning problems, yet most existing approaches rely on fixed group actions or predefined state abstractions. Classical reinforcement learning algorithms typically assume a globally structured Markov decision process with uniformly applicable actions and transitions, an assumption that limits their ability to exploit modularity and local, context-dependent regularities present in many realistic environments. We propose a reinforcement learning framework using groupoids to capture local, state-dependent symmetries and support the dy- namic discovery of equivalence structures during interaction. The agent maintains orbit representatives together with transporters that map raw states to canonical forms, enabling learning and decision-making to be performed in a symmetry-reduced space while preserving local distinctions. Empirical results demonstrate that the proposed groupoid-based approach improves sample efficiency and convergence in dense and large-scale environments exhibiting strong partial symmetries, yielding substantial performance gains over standard Q-learning. These findings show that dynamically exploiting local symmetry provides a practical and mathematically principled route to scalable and generalisable reinforcement learning.

1 INTRODUCTION

Reinforcement learning supports sequential decision-making across domains, but standard methods struggle with experience efficiency, convergence, and generalization when states share local structure. This paper introduces groupoid-based representations to exploit local symmetries, reporting improved learning speed and transfer in dense and large environments with partial symmetries.

1 INTRODUCTION

  • RL is used for sequential decision-making in domains including games, robotics, healthcare, and industrial optimization.The introduction describes agents learning through interaction to optimize long-term cumulative reward.Source: §1 · p. 1
  • Standard RL struggles with inefficient experience use, slow convergence in sparse-reward or high-entropy settings, and generalization across states with repeated local structure.The introduction frames improved efficiency and robustness as ways to reduce reliance on simulation data and computational costs.Source: §1 · p. 1
  • Symmetry-aware methods and disentangled representations use structural regularities to support experience sharing, local generalization, knowledge transfer, and reduced sample complexity.The introduction situates these approaches alongside geometric deep learning, equivariant architectures, and structured representation learning.Source: §1 · p. 1
  • The paper introduces groupoids to represent local symmetries, using orbit representatives and transporters to share experience while preserving distinctions between states.The framework is described as an online-adaptable abstraction that does not require global symmetry.Source: §1 · p. 2
  • The reported contributions are a groupoid-based framework and empirical evidence of faster convergence, lower policy variance, and efficient transfer in environments with repeated structural motifs.The reported benefits are particularly relevant in large or dense settings where classical methods struggle.Source: §1 · p. 2

2 BACKGROUND

This background moves from standard MDP and Q-learning foundations to symmetry-aware representations, contrasting globally acting groups with groupoids for local, context-dependent equivalences.

2.1 Traditional Implementations

  • Traditional RL models sequential agent–environment interaction with MDPs; Q-learning estimates action values recursively, but partial observability or hidden state can make representations intractably large or incomplete.MDPs specify states, actions, transition probabilities, rewards, and a discount factor.Source: §2.1 · p. 2

2.2 Disentangled Representations

  • Disentangled representations factor observations into semantically meaningful components such as position or orientation, supporting information sharing across equivalent states, interpretability, and transfer.The discussion also connects structural representations to compact value functions and policies.Source: §2.2 · pp. 2–3

2.3 Group-based Approaches

  • Global group actions let equivariant or invariant models share parameters across equivalent states, but cannot capture symmetries that are only partial, local, or context-dependent.The paper motivates flexible alternatives because realistic domains may lack one globally consistent symmetry action.Source: §2.3 · p. 3

2.4 Groupoid-based Approaches

  • Groupoids represent states as objects and verified local symmetries as invertible morphisms, preserving neighborhood structure without requiring every transformation to compose with every other.In the RL setting, neighborhood structure includes admissible actions, transition support, and rewards.Source: §2.4 · p. 4

3 GROUPOID-BASED REINFORCEMENT LEARNING

The framework represents local symmetries with canonical state representatives and transporters, then performs learning in canonical space while sharing information only across verified local connections.

3.1 Environmental Representation

  • Groupoid learning replaces global symmetry with local, invertible morphisms that preserve rewards and transitions, and updates canonical representatives along verified connections.Transporters map observed states to their representatives, supporting selective generalisation where symmetry holds only locally.Source: §3.1 · pp. 4–5

3.3 Canonicalisation

  • Stabiliser actions organize context-valid equivalences; representatives are discovered incrementally, with local fingerprints distinguishing repeated intersections without assuming city-wide symmetry.For states with non-trivial stabilisers, value estimates may be averaged over stabiliser actions to enforce invariance and reduce variance.Source: §3.3 · p. 5

3.2 Algorithm

  • The algorithm canonicalizes current and successor states, lifts canonical actions to concrete actions, and creates a representative with an identity transporter when no match exists.Transporters are invertible and compose only when valid, preventing invalid information sharing.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 State Representation

  • A persistent object store records orbit representatives, while transporters map raw states into their canonical frames and can transform action labels to match orientation.Fingerprints and a canonical map link similar local structures to candidate objects or initialise new representatives.Source: §3.4 · p. 6

3.5 Learning Mechanism

  • Updates occur in canonical state-action space, propagating values across verified local transformations while retaining the standard Q-learning update.This reduces redundancy and improves sample efficiency and generalisation where global symmetry assumptions do not hold.Source: §3.5 · p. 6

4 EMPIRICAL EVALUATION

The evaluation tests whether groupoids exploit local structural equivalences in grid worlds where global symmetries are absent. Agents observe a 3 × 3 local window and use rotational and reflectional equivalences; experiments vary environmental density and grid size.

4.1 Impacts of Density and Granularity

  • In dense 200 × 200 environments, the framework maintained over 1,350 local orbits across 40,000 states and outperformed standard Q-learning as complexity increased.Sharing experience across locally equivalent states accelerated policy convergence.Source: §4.1 · p. 7
  • In a 50 × 50 high-density environment, groupoid learning starts slower but rapidly surpasses standard implementations and converges faster.Source: Figure 3 · p. 7
  • Very sparse environments expose a lower boundary for effective abstraction: uniform empty 3 × 3 cells can lead to overgeneralisation and less precise navigation.With few possible orbits, shared learning may lack the granularity needed for precise navigation.Source: §4.1 · p. 7

4.2 Scalability and State Space Compression

  • In grids of 100 × 100 and above, groupoid learning shows superior optimisation to traditional methods, with reward propagation across equivalent orbits after goal discovery.The resulting performance spike is illustrated in Figure 5.Source: §4.2 · p. 8, Figure 5 · p. 8
  • Shared-learning benefits persist in 25 × 25 tasks, although simpler environments reduce the practical advantage.Overall, the framework improves efficiency and generalisation over standard baselines.Source: §4.2 · p. 8

4.3 Orbital Dynamics and Discovery Efficiency

  • Orbit discovery is rapid initially; mapping raw states to canonical orbits yields sub-linear growth in representatives as physical grid cells grow quadratically.The reduced effective state space supports faster policy evaluation, value propagation, and generalisation.Source: §4.3 · p. 8

5 DISCUSSION

The discussion argues that groupoids make reinforcement learning viable under strong local but weak global symmetries, with gains greatest where local structure recurs. It frames the approach as a bridge between local representations and globally symmetric models, while identifying engineering and evaluation work still needed.

5 DISCUSSION

  • Groupoid-based RL reduces effective sample complexity, accelerates convergence, and stabilizes long-run performance in environments with strong local but weak global symmetries.The greatest gains occur where recurring local structure yields more informative orbits for shared learning.Source: §5 · p. 8
  • The method adds engineering overhead that designers must weigh against environment-specific benefits; costs are partly amortized over longer training horizons.Groupoid methods may suit settings that prioritize long-term learning efficiency and generalization over minimal per-step computation.Source: §5 · p. 8
  • Groupoids bridge purely local representations and globally symmetric models, but the present study focuses on discrete, carefully controlled environments.The discussion names robotics, networked systems, and structured simulation as domains with relevant structural regularities.Source: §5 · p. 8
  • Future work should broaden evaluation to realistic, stochastic, and large-scale settings, optimize systems overhead, and adapt modern RL enhancements to groupoid-aware learning.The paper describes these steps as necessary to move from proof of concept to a robust practical alternative.Source: §5 · p. 8
Loading 2609.13035v1…