PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning and interpreting policies for simultaneous entanglement requests in quantum networks

Leon Rode, Sumeet Khatri, Supartha Podder

arXiv:2609.30157v1quant-phcs.LG

TL;DR

Scheduling simultaneous experiment jobs over entanglement-limited networks differs from prior work on virtual-link request queues, and hardware preferences add placement constraints. The paper trains reinforcement-learning policies for this link-layer task; they outperform heuristic baselines in robustness to low link activation probabilities, while an LLM-derived heuristic performs nearly as well as the trained policy.

  • Problem

    Prior approaches addressed virtual-link request queues, while scheduling user-requested experiment jobs with variable topologies and hardware-specific placement preferences remains the task considered here.

    Source: §1 · p. 2

  • Method

    The paper applies reinforcement learning to link-layer experiment scheduling and develops policies whose behavior can be examined through metrics and example trajectories.

    Source: §6.2 · p. 8, §7 · p. 9

  • Results

    Across evaluated settings, DQN policies are more robust to low link activation probabilities; on the dumbbell, the model first fails at 71% lower probability than AGECRITICALFIRST.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The results suggest that learned policy trajectories can help an LLM produce a useful, more interpretable heuristic, with potential relevance to larger networks where direct training is computationally expensive.

    Source: §7 · pp. 9–10

    The model assumes entanglement swapping succeeds deterministically.

    Source: §2 · p. 3

Abstract

from arXiv · show

Future quantum networks will make use of entanglement to perform numerous tasks, such as sending quantum information over long distances, distributed quantum computing, and quantum sensing. In general, these tasks will need to be performed simultaneously in various regions of a network, while minimizing resources and latency. We will thus require policies for scheduling link-level entanglement resources, and using the link-level entanglement to create various forms of multipartite entanglement required for every task. In this work, we address this problem using reinforcement learning. We formulate a Markov Decision Process for the problem and use double deep Q-networks (DQN) with Message Passing Neural Networks (MPNNs), experience replay buffers, and curriculum training to obtain policies. The key physical parameter is the probability of link-level entanglement generation, i.e., the link activation probability. We show that our policies maintain 100% success for up to 71% lower link activation probability than the baseline heuristics for a set of physically relevant network topologies. We then examine an additional constraint where experiment (task) placements are restricted to specific hardware types and demonstrate a similar advantage in performance over heuristics, with our policy maintaining at least an 80% success rate for up to a 59% lower link activation probability. Finally, we explore methods to interpret the learned policy by defining metrics enabling conclusions to be drawn about the model's behavior and by tasking a large language model (LLM) to derive a novel heuristic given example actions taken by the DQN-trained policy. We find that the LLM heuristic performs similarly to the DQN-trained policy in performance, indicating a promising method for interpretable policy extraction for large quantum networks, where direct training becomes computationally expensive.

1 INTRODUCTION

The introduction frames simultaneous experiment scheduling as a link-layer challenge distinct from prior virtual-link request scheduling, with hardware placement constraints as an additional case. It presents reinforcement learning as a way to improve robustness and derive an interpretable heuristic.

1 INTRODUCTION

  • Entanglement swapping can create virtual links between distant nodes when a circuit's required connectivity does not match the network's physical topology.The introduction illustrates the mismatch with a four-node ring and a circuit requiring a link between non-neighboring nodes.Source: §1 · p. 2
  • Prior work studied queues of virtual-link requests, whereas this work schedules experiment jobs represented as network subgraphs to minimize the time needed to generate them.The paper locates virtual-link generation in the connectivity layer and experiment scheduling in the link layer.Source: §1 · p. 2
  • The scheduling problem matters because users may request circuits with variable topology demands on networks constrained by a fixed physical topology.The paper describes this as a practical implication of broader quantum-internet use.Source: §1 · p. 2
  • Hardware-specific placement constraints add a case in which circuits must be scheduled on compatible nodes while maximizing fidelity.The paper motivates such preferences by heterogeneous quantum-memory types.Source: §1 · p. 2
  • The policies outperform heuristics in lossy environments, retain an advantage under node-placement constraints, and yield an LLM-extracted heuristic that performs nearly as well.The comparison uses success rate and mean episode length to assess robustness to link activation probability.Source: §1 · p. 2

2 NETWORK MODEL AND PROBLEM FORMULATION

The model represents probabilistic link activation, virtual-link generation, experiment placement, and resource constraints as a scheduling problem. Its MDP supplies actions and rewards for generating and placing experiments while accounting for topology, link lifetimes, and optional hardware preferences.

2 NETWORK MODEL AND PROBLEM FORMULATION

  • A network is a multigraph with μ parallel sublinks between neighboring nodes; inactive links activate with probability pl = e−γ and remain active for at most m⋆ time steps.Figure 1 illustrates a four-node ring and the aging of active links.Source: §2 · pp. 2–3, Figure 1 · p. 3
  • Entanglement swapping along a connecting path generates a virtual link whose age is determined by the path links, consumes those links, and is assumed to succeed deterministically.The example swaps links of ages 1 and 2 to establish a virtual link of age 3.Source: §2 · p. 3, Figure 1 · p. 3
  • An experiment is a required link topology with a duration, and placement requires a subgraph match using links that remain active for the experiment's duration.Placed links are locked while the experiment runs, then deactivated and returned to the activation process.Source: §2 · p. 3
  • A hardware preference restricts a colored experiment node to a host node with an intersecting color set, while uncolored experiment nodes remain hardware-agnostic.The constraint models placement on nodes with desired memory characteristics.Source: §2 · pp. 3–4

2.1 MARKOV DECISION PROCESS FORMULATION

  • The MDP state records available memories, active-link status and age, locked-link time remaining, and completed experiments; actions place, generate, or wait.A binary action mask indicates valid placements and virtual-link generations.Source: §2.1 · p. 4
  • Virtual-link rewards use subgraph edit distance to penalize unhelpful generations and favor links that reduce the remaining distance to an experiment placement.The reward combines a base amount, a penalty for remaining distance, and a shaping term for progress.Source: §2.1 · p. 4
  • A bottleneck-aware reward uses the maximum betweenness centrality of nodes on the path consumed to generate a virtual link.Betweenness centrality reflects how often a node lies on shortest paths between node pairs.Source: §2.1 · p. 4
  • The reward design offsets larger experiment edge counts, penalizes each time step, and accounts for the complexity of experiment placement.This balances experiment complexity against the objective of minimizing total placement time.Source: §2.1 · p. 5

3 REINFORCEMENT LEARNING FRAMEWORK

The framework trains double deep Q-learning policies for environments with varying link activation noise. Curriculum training and replay buffers support learning across increasing noise, while an MPNN processes graph-structured network information.

3 REINFORCEMENT LEARNING FRAMEWORK

  • Double deep Q-learning trains an online network that interacts with the environment and a target network that evaluates actions.The model minimizes the disagreement between the two networks.Source: §3 · p. 5
  • Training proceeds through 11 curriculum phases from γ = 1.5 to 5.8, carrying successful policy weights and episode roll-outs into progressively noisier conditions.Link activation probability falls from approximately 0.22 to 0.003 across the curriculum; phase mastery requires a 100% mean success rate over 100 episodes.Source: §3 · p. 5
  • A replay buffer combines expert roll-outs with online ε-greedy transitions, and an MPNN supports decisions from graph-structured information.The 100,000-transition buffer supplies mini-batches with 25% expert and 75% online transitions.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

The evaluation uses three nine-node network topologies and compares learned policies with three greedy heuristics. It also tests a colored starlink case in which hardware preferences constrain experiment placement.

4.1 EVALUATION TOPOLOGIES

  • The starlink, dumbbell, and grid evaluations use nine nodes, five parallel sublinks, maximum link age 52, and two K4 experiments, with unsuccessful episodes truncated after 200 steps.The dumbbell models a bridge bottleneck, while the grid represents a denser network.Source: §4.1 · pp. 5–6

4.2 BASELINE HEURISTIC ALGORITHMS

  • The benchmarks are AGECRITICALFIRST, SHORTESTHOPFIRST, and DCTR, greedy heuristics prioritizing link expiry, shortest physical path, and endpoint degree, respectively.AGECRITICALFIRST places the valid experiment whose host links expire soonest; DCTR prioritizes the virtual link with the greatest endpoint-degree sum.Source: §4.2 · p. 6

4.3 COLOR CONSTRAINT

  • The colored starlink test requires red and green experiment nodes to map to matching colored hosts, creating contention between distant hardware preferences and central network resources.Uncolored experiment nodes may map to any topology node, including colored nodes.Source: §4.3 · p. 6, Figure 3 · p. 6

5 RESULTS

Across the tested uncolored and colored topologies, the learned model is more robust to reduced link activation probability than the baseline heuristics. It also completes episodes in fewer steps in the reported operating regimes.

5 RESULTS

  • Evaluations use 25 γ values from 1.5 to 5.8 and 100 independent episodes to calculate success rate and mean step count.Appendix E describes how P was selected for each topology by comparing performance after each curriculum-training phase.Source: §5 · p. 6

5.1 MODEL VS BASELINES ON UNCOLORED STARLINK TOPOLOGY

  • On the starlink topology at P = 9, the model first fails at γ ≈ 4.5 (pl ≈ 0.011), 59% below AGECRITICALFIRST and 66% below DCTR; at γ = 4.0 (pl ≈ 0.018), it places experiments in ≈95 steps versus all heuristics’ ≈170.Source: §5.1 · p. 6

5.2 MODEL VS BASELINES ON UNCOLORED DUMBBELL AND GRID TOPOLOGIES

  • On dumbbell, the model first fails at 66% lower probability than DCTR and 71% lower than AGECRITICALFIRST; on grid, the corresponding differences are 51% versus SHORTESTHOPFIRST and 59% versus AGECRITICALFIRST.On dumbbell, the model also has a strictly lower mean step count across the operational regime; at γ = 4.0 it takes approximately 50 steps versus 125–150 for heuristics, while on grid it takes around 55 versus 125–160.Source: §5.2 · pp. 7–8, Figure 5 · p. 7

5.3 MODEL VS BASELINES ON COLORED STARLINK TOPOLOGY

  • On colored starlink, the model begins to fail more than 20% of episodes at 51% lower probability than AGECRITICALFIRST and 59% lower than DCTR.For γ < 4.1, it completes episodes at least 60 steps quicker than the most efficient heuristic.Source: §5.3 · p. 8, Figure 6 · p. 8

6 BEHAVIOR PROFILING

The paper profiles policy behavior with three metrics and asks an LLM to derive a heuristic from DQN-policy action examples. The resulting heuristic captures a two-hop requirement, while DQN and LLM differ in robustness and placement patterns.

6.1 LLM POLICY DISTILLATION

  • The LLM-derived policy partitions nodes into hubs and leaves, prioritizes hub placements, and generates virtual links when no experiment can be placed.DQN succeeds more frequently for γ values below approximately 5.0 and completes episodes in fewer steps for γ > 3.2.Source: §6.1 · p. 8, Figure 7 · p. 9

6.2 METRICS

  • HOLDINGTIME measures active unused sublink time, BRIDGESPAN measures generated virtual-link path length, and HUBANCHORBIAS measures the maximum endpoint degree.On starlink, DQN has greater HOLDINGTIME than heuristics, generates shorter-span links in the operational regime, and has lower HUBANCHORBIAS; the LLM has BRIDGESPAN of two and HUBANCHORBIAS of four.Source: §6.2 · pp. 8–9

7 CONCLUSION

The work applies reinforcement learning to link-layer scheduling under realistic constraints and explores LLM-based policy extraction from model trajectories. Its results support reliable operation at low link activation probabilities and suggest LLMs can produce useful, naturally described policies.

7 CONCLUSION

  • The learned policy operates reliably at low link activation probabilities, while LLMs can derive useful, naturally described policies from its trajectories.The conclusion frames these results as potentially aiding near-term network topology design and end-user request management.Source: §7 · pp. 9–10
  • Figure 8 tracks HOLDINGTIME, BRIDGESPAN, and HUBANCHORBIAS for the model, baseline heuristics, and LLM heuristic on successful uncolored starlink episodes.Absent points indicate a policy had a 0% success rate at that γ value.Source: Figure 8 · p. 9
  • The conclusion presents LLM policy extraction as a complement to targeted behavior metrics when learned policies are difficult to interpret.The authors suggest LLMs may extract policies that are both easily described and useful in more general cases and larger networks.Source: §7 · pp. 9–10

8 AI USE DISCLOSURE

This material discloses AI use and reproducibility resources, then describes double deep Q-learning and a graph-based observation and message-passing architecture for scheduling decisions.

8 AI USE DISCLOSURE

  • The study used Gemini 3.1 Pro for an initial MPNN and training-loop implementation, experiments, plotting code, and an initial Figure 1, with authors manually reviewing and revising the implementation.The authors state that AI did not interpret results or design methodology or experiments.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

  • Training and evaluation code, environment configurations, and repository instructions are provided for reproducibility.The repository README contains instructions.Source: REPRODUCIBILITY STATEMENT · p. 10

A DOUBLE DEEP Q-LEARNING

  • The online network selects actions through ε-greedy exploration, while replay-buffer transitions train Q-value estimates using target-network evaluations.The online network selects the best next-state action; the target network evaluates it, and the online estimate is trained against the resulting temporal-difference target.Source: A DOUBLE DEEP Q-LEARNING · p. 14
  • Target-network parameters receive Polyak soft updates at each training step to keep target Q-values changing smoothly.The passage states that the small τ is intended to avoid moving-target instabilities.Source: A DOUBLE DEEP Q-LEARNING · p. 14

B OBSERVATION SPACE

  • The observation represents normalized node memory availability, physical-link age and locked-link remaining time, plus which experiments remain unplaced.Scalar features are normalized to [0, 1].Source: B OBSERVATION SPACE · p. 14

C MESSAGE PASSING NEURAL NETWORK

  • An MPNN processes graph-structured observations using learned messages between neighboring nodes and node-state updates.The message function uses a two-layer MLP with ReLU activations.Source: C MESSAGE PASSING NEURAL NETWORK · p. 14

2 ReLU

  • After message passing, mean-pooled node states are combined with global experiment status to produce action Q-values.The action head heavily penalizes illegal actions by modifying their Q-values to −10^9.Source: §2 · p. 15

D PERFORMANCE ON COLORED DUMBBELL AND GRID TOPOLOGIES

On colored dumbbell and grid topologies, the model is evaluated under resource contention created by experiment placements. Its success-rate advantage over heuristics differs across the two topologies.

D PERFORMANCE ON COLORED DUMBBELL AND GRID TOPOLOGIES

E.1 UNCOLORED VARIANT

Later curriculum phases show deterioration in low-noise starlink performance, despite earlier phases succeeding perfectly there.

E.1 UNCOLORED VARIANT

  • On uncolored starlink, phases P ≤ 9 succeed perfectly at low γ, but later phases deteriorate as high-noise experience overrides earlier learning.The comparison covers all 11 curriculum phases on starlink, dumbbell, and grid topologies.Source: §E.1 · p. 16

E.2 COLORED VARIANT

For the colored variant, earlier curriculum phases were more robust to noise than later phases. The most robust stopping phases differed by topology.

E.2 COLORED VARIANT

  • Unlike the uncolored case, early curriculum phases were more robust than later phases in the colored variant.For uncolored starlink, dumbbell, and grid topologies, the most robust stopping phases were P = 9, P = 8, and P = 8, respectively.Source: §E.2 · p. 16, Figure 11 · p. 17
  • The colored-variant comparison covers 11 curriculum phases on each topology.Figure 12 compares robustness to changes in link activation probability across those phases.Source: §E.2 · p. 16, Figure 12 · p. 18
  • P = 7, P = 6, and P = 7 provided the greatest robustness to noise for the colored starlink, dumbbell, and grid topologies, respectively.Figure 12 compares robustness across curriculum phases for each topology.Source: §E.2 · p. 16, Figure 12 · p. 18

F LLM PROMPT

The LLM prompt uses DQN execution traces to request pseudocode for a general heuristic for starlink networks with three hubs and varying numbers of leaves. The resulting heuristic performs well across the tested network sizes, and the paper reports that it can be reused as the network expands.

F LLM PROMPT

  • The prompt asks an expert policy extractor to derive a general heuristic from execution traces for a starlink with N=9 nodes and three hubs, applicable to any number of evenly distributed leaves.The prompt requests pseudocode as the output.Source: F LLM PROMPT · p. 19
  • The prompt's stated environment has five parallel physical links, each activated per time step with probability p_link = 0.0111.Virtual links are deterministically generated by consuming physical links along shortest paths and persist for m* before deactivating.Source: F LLM PROMPT · p. 19
  • The traces let the LLM infer timing between virtual-link generations, map endpoint indices to the host network using its adjacency matrix, and inspect action consequences through rewards and episode results.An excerpt is from the first episode.Source: F LLM PROMPT · p. 19
  • The LLM-generated heuristic performs well independently of network size as the uncolored starlink topology expands.The tests keep three central hub nodes fixed and distribute additional nodes evenly among peripheral nodes; Figure 13 compares success rate and mean episode length across topology sizes.Source: F LLM PROMPT · p. 20, Figure 13 · p. 20

Related papers

Loading 2609.30157v1…