PaperlayerView exact revision on arXiv

Source-linked AI summary

CDRL: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery

Piyush Jha, Jake Rudolph, Victoria Knapp-Pérez, Max Fieg, Aishik Ghosh, Vijay Ganesh

arXiv:2608.20686v1cs.AIcs.LGcs.LOhep-ph

TL;DR

Scientific model discovery must search enormous constrained hypothesis spaces, but scalar-reward RL gives limited information about why candidates fail. CDRL converts verifier-generated certificates into reusable constraints and improves neutrino-model discovery across three theory spaces while also extracting reusable search rules.

  • Problem

    RL methods typically rely on scalar rewards that provide limited information about candidate failure, despite scientific tasks offering structured feedback from external reasoning tools.

  • Method

    CDRL converts certificate-identified failure assignments into symbolic conflict clauses enforced during search, allowing partial candidates matching known invalid patterns to be eliminated.

  • Results

    Across three theory spaces, CDRL achieves up to 1.95× higher valid model rates and 6.33× higher neutrino model rates than AMBer while evaluating up to 4× fewer candidates.

  • Takeaways & Limitations

    CDRL uncovers reusable structure in constrained combinatorial search through both certificate-derived constraints and interpretable rules extracted from search trajectories.

  • Takeaways & Limitations

    CDRL depends on useful certificate-generating tools, is currently best suited to structured discrete spaces, and may incur certificate-analysis and clause-management overhead.

Abstract

from arXiv · show

Many scientific discovery problems require searching combinatorial hypothesis spaces under complex domain constraints. Reinforcement learning (RL) offers a promising approach, but existing methods rely on scalar rewards that provide limited information about why candidate solutions fail, leading agents to repeatedly explore invalid regions. We introduce Certification-Driven Reinforcement Learning (CDRL), a framework that leverages structured feedback from symbolic reasoning tools. When a candidate violates domain constraints, these tools produce certificates identifying the actions responsible for failure. CDRL converts these certificates into reusable constraints that eliminate classes of invalid solutions and guide exploration toward valid regions. We evaluate CDRL on neutrino flavor model discovery in theoretical particle physics, where the hypothesis space exceeds $10^{26}$ possible models, and compare it with the state-of-the-art RL approach previously used for this task. Across three theory spaces, CDRL achieves up to 1.95$\times$ higher valid model rates and up to 6.33$\times$ higher neutrino model rates while evaluating up to 4$\times$ fewer candidates. We further extract 40 interpretable rules from search trajectories using a post-hoc decision-tree framework and show that reusing them as soft constraints yields gains of up to 2$\times$ in valid model rates and 3$\times$ in neutrino model discovery across all three theory spaces. These results suggest that CDRL uncovers reusable structure in combinatorial search spaces and provides a general framework for scientific model discovery.

1 Introduction

Scientific discovery often requires searching enormous combinatorial hypothesis spaces, while conventional RL provides limited information about why candidates fail. CDRL combines scalar rewards with certificates from symbolic reasoning tools to prune invalid regions and improve neutrino-model discovery.

  • Motivation: Scientific discovery spans combinatorial hypothesis spaces where small structural changes can produce dramatically different predictions, making exhaustive exploration computationally challenging.The motivation covers particle physics and other scientific domains.
  • Motivation: RL constructs candidate models sequentially but typically relies on scalar rewards that provide little information about failure causes.This limitation can lead agents to revisit invalid regions.
  • Motivation: Structured certificates identify the assignments responsible for violated constraints, complementing rewards by providing reusable hard feedback for search.External reasoning tools can generate these certificates from formally defined candidate objects.
  • Application: The neutrino flavor task contains more than 10^26 possible models, with each candidate requiring non-trivial symbolic and numerical evaluation.Models must satisfy domain constraints and reproduce experimentally relevant properties.
  • CDRL approach: CDRL converts failure certificates into symbolic conflict clauses that a SAT-based referee enforces during search, eliminating equivalent invalid patterns before physics evaluation.The system uses Boolean Constraint Propagation to block partial assignments that repeat known conflicts.
  • Results: 1.95× higher valid model rates and 6.33× higher neutrino model rates were achieved than AMBer while evaluating up to 4× fewer candidates.The comparison spans three theory spaces.
  • Results: Post-hoc decision-tree analysis extracted 40 interpretable rules from search trajectories, whose reuse as soft constraints further improved performance.The rules capture recurring dependencies between particle assignments.

2 Background and Related Work

Neutrino flavor model construction addresses the origin of neutrino masses, fermion flavors, and observed mixing within a constrained particle-physics framework. The paper situates CDRL among RL, neuro-symbolic, oracle-augmented, and scientific-discovery methods while specifying its algorithmic components and a VEV assumption.

  • Neutrino flavor models: Neutrino masses require particles beyond the Standard Model, linking model construction to the origin of fermion flavors and observed mixing.New particles coupling to neutrinos are generally expected to couple to charged leptons as well.
  • Neutrino flavor models: Flavor models combine particle content, symmetry representations, Abelian charges, and vacuum expectation values to structure predictions.The framework aims to constrain flavor-sector parameters through global flavor symmetries.
  • Related work: AMBer uses domain-specific physics tools to evaluate candidate models one at a time, whereas CDRL extracts certificates from those tools and reuses them during search.This positions CDRL as an extension of prior RL-based particle-physics model search.
  • Related work: Prior work applies AI and RL to scientific discovery, algorithm discovery, physical control, and experimental design across diverse domains.Examples include materials discovery, biomolecular structure prediction, tensor decomposition, and quantum circuits.
  • Related work: CDRL follows a modular neuro-symbolic route in which a learning agent queries external symbolic systems for structured candidate feedback.This connects the framework to oracle-augmented learning and SAT-style conflict feedback.
  • Scope assumption: The setup assumes that a specified vacuum-expectation-value pattern can be achieved rather than solving its dynamical origin.The paper identifies achieving the VEV pattern as a separate problem.
  • Algorithm: The CDRL algorithm initializes constraints, plans candidates with a policy-value network, verifies them, stores valid hypotheses, and converts failure certificates into accumulated constraints.The notation distinguishes the policy πθ, verifier V, analyzer A, initial constraints K0, and accumulated constraints K.

3 Certification-Driven Reinforcement Learning (CDRL)

CDRL augments sequential hypothesis search with certificate-based constraint learning. Rejected candidates yield reusable constraints that accumulate in the planner and restrict future exploration to configurations consistent with discovered knowledge.

  • Framework: CDRL views scientific discovery as structured search over candidate hypotheses verified against domain constraints.The agent incrementally constructs candidates while interacting with an external verifier.
  • Certificate learning: When verification rejects a candidate, CDRL converts its certificate into a reusable constraint that rules out similarly invalid candidates.The learned constraint targets a class of configurations rather than only the rejected instance.
  • Constraint accumulation: The agent maintains a growing constraint set K, initialized with K0, and uses PLAN(πθ, K) to construct candidates while enforcing those constraints.Planning may use MCTS guided by the policy-value network.
  • Constraint accumulation: Accumulated constraints progressively restrict exploration, helping the agent avoid rediscovering similar failure modes and focus on more promising configurations.This restriction is applied as training proceeds.

4 Application: Neutrino Flavor Model Discovery

CDRL applies structured symbolic constraints and certificate-derived conflict clauses to neutrino flavor model discovery, combining them with MCTS to prune invalid model classes and guide search. The approach evaluates candidate theories through particle-physics pipelines while enforcing consistency during construction.

  • Problem setup: The task constructs neutrino flavor models from symmetry choices, particle representations, Abelian charges, and vacuum expectation values in a large combinatorial space.Candidate theories are encoded as matrix representations of particle assignments and charges.
  • Sequential construction: The agent assigns model components sequentially in logical blocks, beginning with the active ZN symmetry and ending with flavon VEV configurations.Particle representations and Abelian charges are assigned as separate blocks for each particle.
  • Constraint-guided search: A symbolic constraint layer uses Boolean constraints and BCP to reject inconsistent partial assignments and infer logically forced assignments during MCTS.The constraints include representation, charge, triplet-structure, active-symmetry, and VEV requirements compiled into CNF.
  • Evaluation and search: At terminal states, the physics pipeline evaluates fully specified models, and CDRL uses MCTS with policy-value guidance and constraint-masked actions to explore them.Physics evaluation produces rewards after symbolic and numerical processing, while conflict detection removes assignments that trigger learned clauses.
  • Certificate-driven pruning: Certificate analysis converts failure-causing assignments from invalid terminal models into reusable conflict clauses that prune entire classes of candidates.The clauses are added to a global constraint set and can be applied before subsequent search episodes.
  • Certificate types: The certificate types target insufficient-rank structures and Abelian-charge equivalences, including reducibility and charge negation, while preserving unaffected alternatives.Representation-and-VEV certificates can block combinations that cannot achieve full rank under any charge assignment.
  • Search behavior: Removing constraint-related sharing increases variance and reduces neutrino discovery by up to 2×.This result is reported as an additional implementation study in Table 5.

5 Experimental Results

CDRL improves neutrino flavor model discovery across theory spaces through certificate-driven pruning, efficient evaluation, and reusable search rules. Its discovered models also cover broader symmetry and lower-complexity regions, although rule interpretation remains limited by selective trajectories and added computational overhead.

  • Main Results: 26.9% valid models versus 13.79% for AMBer on A4 × ZN, a 1.95× improvement, while neutrino models rise from 0.03% to 0.19%, a 6.33× improvement.Similar gains occur elsewhere, including up to 1.65× higher valid-model rates and 3× higher neutrino discovery.
  • Main Results: Up to 4× fewer candidate evaluations still yield more discovered neutrino models than AMBer and random search.Candidate evaluations are used as the normalized efficiency measure because physics evaluation dominates computational cost.
  • Qualitative Trends: CDRL spreads viable models across a broader range of ZN symmetry groups, whereas AMBer concentrates discoveries around specific values such as N = 5.The reducibility certificate prevents redundant representatives from being explored and counted under different symmetry groups.
  • Ablation Study: Removing symbolic certificates lowers A4 × ZN valid-model rates from 26.9% to 16.2% and neutrino discovery from 0.19% to 0.03%.Removing the neural network or MCTS collapses neutrino discovery to near zero, while portfolio sharing affects performance and variance.
  • Qualitative Trends: CDRL places more discovered-model density in low-np regions containing well-fitting, low-complexity models.Very low χ2 differences do not imply meaningfully different fit quality, so Tables 3 and 5 provide the primary quantitative comparison.
  • Search Knowledge Discovery: Rule-guided runs improve valid-model rates by roughly 1.5–2× and neutrino discovery by 2–3× across three theory spaces.The rules are extracted from high-confidence MCTS trajectories and injected as soft constraints, but these smaller-scale results are not directly comparable to full-scale Table 3 results.
  • Search Knowledge Discovery: Decision trees recover recurring assignment patterns, including coordinated charged-lepton-singlet and triplet assignments, but may reflect search-process frequencies rather than physical correlations.Each tree is fit to a small, selective high-confidence sample, so apparent rules may not characterize the wider search.

6 Conclusion, Limitation, and Future Work

CDRL combines certificate-driven pruning with reinforcement learning to improve scientific model discovery, while extracted rules provide interpretable but not yet physically validated search knowledge.

  • Conclusion: CDRL converts symbolic failure certificates into reusable constraints, improving search efficiency in constrained hypothesis spaces.The framework identifies why candidates fail and reuses those failures to prune subsequent exploration.
  • Conclusion: 1.95× higher valid model rates and 6.33× higher neutrino model discovery are achieved per candidate evaluated compared with AMBer.Comparable numbers of neutrino models require up to 4× fewer physics evaluations.
  • Conclusion: CDRL improves theory-space coverage, matches AMBer’s best model complexity, and discovers more models while extracting interpretable rules from trajectories.The rules can be reused as soft constraints in later searches.
  • Future Work: The framework’s generality is motivated by scientific domains combining large discrete spaces, expensive verifiers, and tools that explain constraint violations.The paper names chemistry, materials science, and automated theorem proving as potential settings.
  • Limitation: CDRL depends on useful certificate-generating tools and is best suited to structured discrete spaces where constraints can be represented and reused.Certificate analysis and clause management may also add overhead in larger search spaces.

root

The CDRL loop masks illegal actions before neural ranking, evaluates selected candidates, and turns failure certificates into clauses that prune matching patterns in other branches.

  • Propose, mask, descend: At every visited node, BCP masks children violating learned clauses before the policy-value network ranks the remaining legal actions.This process repeats across symmetry choices, representations, charges, and subsequent decisions.
  • Two signals from one evaluation: A completed failing candidate produces both a scalar reward for network retraining and a certificate converted into a clause in the shared database.Valid candidates produce a reward alone.
  • MCTS behavior: MCTS selects the move visited most often across simulations rather than simply choosing the network’s top-ranked action.Network updates occur periodically from replay-buffer mini-batches.
  • Same pattern, different branch: A clause learned from one branch permanently prunes a matching representation-and-VEV pattern elsewhere before physics evaluation.The worked example shows the learned clause masking the same pattern in a sibling ZN branch.

B.1 Compute Infrastructure

Experiments use CPU workers for parallel self-play and an H100 GPU for training, while physics evaluation dominates the computational cost.

  • Infrastructure: Parallel self-play runs across 32 CPU cores, while neural-network training uses an NVIDIA H100 GPU with an 80GB memory specification.The reported machine includes Intel Xeon Gold 6448Y CPUs and 32 GB RAM.
  • Computational cost: Physics evaluation dominates computation because it combines symbolic construction, numerical optimization, and parameter fitting.MCTS traversal and SAT-based propagation add overhead but are comparatively lightweight.
  • Computational cost: CDRL uses 32 CPU cores, whereas AMBer uses 250 parallel environments, making compute configurations different across methods.The paper notes these differing parallelization settings when discussing comparisons.

B.2 Additional Implementation Details

The implementation combines SAT-based hard-constraint propagation with neural and planning baselines, and evaluates generated candidates through the shared symbolic and physics pipeline.

  • Constraint reasoning: Domain constraints are compiled into CNF, and MiniSAT-style BCP propagates implied assignments after each action.This keeps intermediate search states consistent with hard constraints.
  • Rule extraction: Search-knowledge trees use high-confidence decisions above 95% dominant-action visits with positive value estimates, minimum sample thresholds, and maximum depth 3.Trees use an 80/20 train-test split when enough data is available.
  • Baselines: AMBer refines models using scalar rewards from fit quality, free-parameter counts, and higher-order ZN exploration bonuses, without certificate-driven pruning.This provides the principal RL comparison baseline.
  • Baselines: Additional baselines isolate planning, neural guidance, and symbolic reasoning through pure MCTS, DNN-only, SAT-based, and DNN+MCTS methods.These comparisons separate the contributions of the main search components.
  • Prompting baseline: The prompting baseline asks GPT-5.3 to autoregressively generate neutrino model matrices from task instructions, formatting constraints, and three valid examples.Temperature is 0.8 and top-p is 0.95; outputs are evaluated through the same symbolic and physics pipeline.

B.4 Neural Network Architecture

The implementation combines a policy-value network with MCTS, symbolic clause propagation, parallel workers, and trajectory-derived rules to improve search efficiency.

  • Neural Network Architecture: The policy-value network uses two fully connected layers, followed by separate policy and value heads.The hidden layers have 512 and 256 units with BatchNorm and ReLU; the policy head uses log-softmax and the value head uses tanh.
  • Training Configuration: Training uses 100 episodes per iteration, 150 MCTS simulations per move, and 20 training epochs per iteration.The policy head uses cross-entropy against MCTS visit counts, while the value head uses mean squared error against normalized terminal rewards.
  • Checkpoint Evaluation: After each iteration, a new checkpoint replaces the previous one only when it achieves higher cumulative terminal reward in fixed self-play evaluation.Both checkpoints independently generate candidates through the same environment and evaluation pipeline.
  • Parallelization: Parallel workers share replay buffers and learned conflict clauses, while cached terminal evaluations avoid redundant computation.Conflict clauses discovered by one worker are synchronized globally and reused by all workers.
  • Efficiency and Scope: The combined parallel search and clause sharing improve exploration and sample efficiency, while learned clauses accumulate monotonically without CDCL-style deletion.The implementation remained manageable in the studied theory spaces because physics evaluation dominated runtime; clause forgetting is proposed for larger domains.
  • Trajectory Analysis: Post-hoc decision-tree analysis of high-confidence MCTS states extracts 40 interpretable rules describing recurring particle-assignment dependencies.These rules are reused as soft constraints to improve subsequent search.

C.1 A4 × Z4

This section reports decision-tree observations for the A4 × Z4 theory space, including charge and representation assignments with their train and test accuracies.

  • Charge assignments: E3.Z4_charge achieves 0.972 train accuracy and 0.944 test accuracy.The reported observation uses E1.Z4 = 1? as its condition.
  • Charge assignments: N1.Z4_charge achieves 0.911 train accuracy and 1.000 test accuracy.The reported observation uses E3.Z4 = 1? as its condition.
  • Representation assignments: The reported representation observation for Hd.rep has 0.615 train accuracy and no test value.Its condition is shown as Hu.Z4 = 4?.
  • Charge assignments: phi1.Z4_charge achieves 0.654 train accuracy and 0.429 test accuracy.The supplied passages report this charge-assignment result without an accompanying condition in the same row.
  • Charge assignments: phi2.Z4_charge achieves 0.719 train accuracy and 0.444 test accuracy.The supplied passages report this charge-assignment result without an accompanying condition in the same row.

C.2 A4 × ZN

This section reports decision-tree observations for the A4 × ZN theory space, covering representation and charge assignments with train and test accuracies, including several high-performing rules.

  • Charge assignments: E2.Z5_charge achieves 0.938 train accuracy and 0.882 test accuracy.The associated condition is E1.Z5 = 2?.
  • Charge assignments: E3.Z5_charge achieves 0.926 train accuracy and 0.968 test accuracy.The associated condition is E1.Z5 = 2?.
  • Charge assignments: N1.Z5_charge achieves 0.861 train accuracy and 0.900 test accuracy.The associated condition is E2.Z5 = 1?.
  • Representation assignments: N1.rep achieves 0.960 train accuracy and 0.932 test accuracy, while E2.rep achieves 0.950 and 0.905.The reported conditions are N3.rep = 3_2b? and E2.rep = 1”, respectively.
  • Representation assignments: phi1.rep achieves 0.859 train accuracy and 0.941 test accuracy, whereas phi2.rep achieves 0.750 and 0.625.The supplied observations include conditions involving E1.rep = 1? and E1.rep = 1”? for these representation assignments.
Loading 2608.20686v1…