PaperlayerView exact revision on arXiv

Source-linked AI summary

Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks

Dang-Dung Vu

arXiv:2608.29256v1cs.NIcs.LG

TL;DR

Online sensing-session consolidation must coordinate reuse, admission, sensing SLAs, communication QoS, and future commitments. CT-PPO trains a factorized constrained policy with same-trace replicas and leave-one-out return contrasts; across matched evaluations, it outperforms JC-PPO and retains an advantage across tested arrival loads.

  • Problem

    Online consolidation must decide whether arriving requests reuse or create sessions, defer, or be rejected while preserving service commitments and managing finite sensing and communication resources.

    Source: §I · p. 1

  • Method

    CT-PPO trains a factorized constrained actor using stochastic replicas on the same workload trace, leave-one-out Monte Carlo reward contrasts, and factor-specific constraint credit.

    Source: §I · p. 2

  • Results

    CT-PPO gains 0.934 macro return over JC-PPO (95% CI [0.702, 1.164]) across five seeds and matched external workloads, and retains a positive return advantage at all three tested arrival loads.

    Source: §VIII · p. 13

  • Takeaways & Limitations

    Deployment uses public observations and hard masks, and CT-PPO's deployed encoder–actor parameter count matches JC-PPO's with effectively unchanged actor-only CPU latency.

    Source: §I · p. 2, §VIII · p. 13

    Evidence is limited to a centralized single-cell monostatic setting and five training seeds; robustness to different task mixes, mobility/channel statistics, sensing-model mismatch, and qualitatively different arrival processes remains untested.

    Source: D. Scope and Limitations · pp. 12–13

Abstract

from arXiv · show

Integrated sensing and communication (ISAC) networks can serve compatible requests through shared sensing sessions, but consolidation couples admission, reuse, profile selection, sensing service-level agreements (SLAs), communication quality of service (QoS), and future commitments. We formulate this problem as a constrained Markov decision process and propose Common-Trace Factorized Constrained Proximal Policy Optimization (CT-PPO). During training, stochastic policy replicas share the same primitive workload trace; leave-one-out discounted Monte Carlo return contrasts provide reward credit to applicable actor factors, while constraint credit remains factor/prefix-specific. Across five training seeds and matched workloads, CT-PPO achieves the highest mean macro return, exceeding matched Joint-Credit PPO (JC-PPO) by 0.934 (95% confidence interval [0.702, 1.164]) and SLA-Aware Greedy by 1.847; versus JC-PPO, it reduces sensing-resource cost by 6.277 and raises accepted requests per created session by 0.0806. A four-way ablation shows that the factorized surrogate alone yields no detectable macro-return gain, whereas adding common-trace reward credit produces the dominant improvement. Without retraining, CT-PPO retains a return advantage at low, nominal, and high arrival loads, with the strongest gain under clustered arrivals. Deployment uses public observations and hard masks; CT-PPO's extra parameters are training-side, its actor footprint matches JC-PPO, and actor-only CPU latency is effectively unchanged.

I. INTRODUCTION

Online sensing-session consolidation must coordinate service commitments and finite sensing and communication resources as requests arrive. The paper formulates this as a constrained decision problem and introduces CT-PPO, whose common-trace reward credit is the main empirical driver of its return gain.

I. INTRODUCTION

  • CT-PPO gains 0.934 macro return over JC-PPO (95% CI [0.702, 1.164]) and 1.847 over SLA-Aware Greedy across five seeds and matched external traces.Against JC-PPO, it also reduces sensing-resource cost by 6.277 and raises accepted requests per created session by 0.0806.Source: §I · p. 2
  • The paper formulates stateful consolidation as a service-level CMDP with MERGE, CREATE, DEFER, and REJECT actions, sensing SLAs, communication QoS, and future commitments.The controller selects service/session decisions and discrete sensing profiles; communication scheduling and continuous physical-layer variables are not policy decisions.Source: §I · pp. 1–2
  • CT-PPO trains stochastic policy replicas on the same exogenous workload trace and uses leave-one-out Monte Carlo return contrasts for applicable action factors.Constraint credit remains factor-specific, while deployment uses public observations and hard feasibility masks without future workload information or oracle outcomes.Source: §I · pp. 1–2
  • The four-way ablation finds no detectable macro-return gain from the factor-wise surrogate alone; common-trace reward credit is the dominant driver of the observed gain.The additional prefix-critic machinery yields no detectable incremental return effect in the evaluated configuration.Source: §I · p. 2

A. Network-Level ISAC Services and Orchestration

  • Prior work covers network-level sensing services and request aggregation, but not the online stateful choice to reuse, create, defer, or reject while preserving member commitments.The cited background describes sensing-as-a-service abstractions and resource optimization or aggregation in related systems.Source: §I · p. 1, A. Network-Level ISAC Services and… · p. 2

B. Constrained and Set-Structured Policy Learning

  • Hard masks enforce physical and contractual feasibility, while long-term sensing SLA and communication QoS residuals remain explicit constrained-policy objectives.The matched JC-PPO comparison shares the Set representation, public features, masks, and conditional action distribution, focusing comparison on credit construction.Source: B. Constrained and Set-Structured… · pp. 2–3

C. Structured Credit Assignment and Matched Rollouts

CT-PPO addresses credit assignment under conditional actions and stochastic exogenous events by matching policy replicas on a complete workload trace. The system semantics clarify that a shared session produces one physical update whose sensing service and resource cost affect multiple tenants and communication users.

C. Structured Credit Assignment and Matched Rollouts

  • Matched-trace replicas fix exogenous arrivals and physical processes while actions generate different endogenous trajectories; slot-aligned leave-one-out returns provide common reward credit to applicable factors.Constraint credit remains specific to the applicable factor; replicas need not remain in the same endogenous state.Source: C. Structured Credit Assignment and… · p. 3
  • Related credit methods include action-dependent baselines, counterfactual action baselines, and common-random-number rollout comparisons, whereas CT-PPO matches complete primitive workload traces.CT-PPO does not enumerate alternative action components or condition credit on learned future-event information.Source: C. Structured Credit Assignment and… · p. 3

A. System and Shared-Session Semantics

  • One scheduled session update produces one shared sensing result, which each member evaluates against its own task, quality threshold, freshness bound, and service interval.A feasible CREATE or MERGE causes a current-slot update, with subsequent updates recurring at the selected profile's period.Source: A. System and Shared-Session Semantics · p. 3, Figure 1 · p. 4

B. Shared Resources and Service Quality

C. Compatibility, Actions, and Service Accounting

A merge is feasible only when it preserves every existing and incoming member's service requirements and deterministic commitments. The action interface then separates masked feasibility from learned preference and tracks service outcomes after admission.

C. Compatibility, Actions, and Service Accounting

  • A feasible merge must preserve target identity, area coverage, task capability, permissions, timing, freshness, quality, and deterministic reservations for every member.Freshness requires the profile update period to respect each member's maximum valid-output age.Source: C. Compatibility, Actions, and Service… · p. 4
  • The feasibility checks cover current merge or create conditions, while unknown future arrivals are not reserved.The cited passage names merge and create checks but does not extend reservations to unknown requests.Source: C. Compatibility, Actions, and Service… · p. 4
  • The action set includes MERGE and CREATE for admission, DEFER with a fixed cooldown, and REJECT; hard masks encode feasibility, not value or heuristic preference.The mask excludes learned reward, cost, or preference from action validity.Source: C. Compatibility, Actions, and Service… · pp. 4–5
  • A first service violation is absorbing, and rejected or expired requests receive no completion value.The request remains attached through its service end after a first violation.Source: C. Compatibility, Actions, and Service… · p. 5

D. Reward, Constraints, and CMDP

The CMDP distinguishes completed-service value and sensing expenditure from long-term sensing SLA and communication QoS constraints. The policy acts on a public observation with current information, while feasibility conditions remain hard-masked.

D. Reward, Constraints, and CMDP

  • The slot reward represents completed value versus sensing cost, with bandwidth and power cost weights and a sensing-cost weight.The supplied passage provides the cost-weight and reward framing, without the full equation.Source: D. Reward, Constraints, and CMDP · p. 5
  • Consolidation has no merge, acceptance, deferral, fairness, or compatibility bonus; it changes return through completed service and sensing expenditure.Action validity, long-term service constraints, and the service-versus-cost objective are explicitly distinct.Source: D. Reward, Constraints, and CMDP · p. 5
  • The formulation gives tenants violation-rate budgets and communication users minimum-rate requirements, with latest-start deadlines as hard pre-admission conditions.Communication shortfall is controlled by a long-term residual rather than a hard mask.Source: D. Reward, Constraints, and CMDP · p. 5
  • The observation includes current physical processes, request and session state, commitments, and cumulative accounting, but excludes future arrivals and other future exogenous realizations.The public observation also includes compatibility and quality margins and hard masks.Source: D. Reward, Constraints, and CMDP · p. 5

A. Shared Policy Structure and Matched Reference

CT-PPO and JC-PPO share the same observations, masks, Set representation, action distribution, and rollout infrastructure. The policy uses a masked factorized actor whose conditional branches follow the environment’s action hierarchy.

A. Shared Policy Structure and Matched Reference

B. Common-Trace Reward Credit

CT-PPO uses shared primitive workload traces for reward credit and hierarchical prefix values for factor-specific constraint credit. The two mechanisms apply to different parts of the actor update.

B. Common-Trace Reward Credit

  • Replicas share a primitive workload trace but draw their own actions, so their endogenous trajectories can diverge while reward comparisons control for the shared exogenous realization.Source: B. Common-Trace Reward Credit · p. 6
  • Decision credits compare discounted slot-return suffixes with leave-one-out peer baselines, aligning replicas by physical slot rather than by state or action sequence.Source: B. Common-Trace Reward Credit · p. 6

C. Factor-Specific Constraint Credit

D. Factorized Constrained PPO Update

The actor update uses factorized reward and constraint surrogates, while evaluation compares policies under matched settings and workloads. Checkpoint selection uses validation workloads before final external evaluation.

D. Factorized Constrained PPO Update

A. Experimental Setting and Online References

  • The primary comparison and component ablation use the frozen nominal environment, while arrival-load stress changes only the specified arrival-rate scalars.Source: A. Experimental Setting and Online… · p. 7
  • Clustered arrivals share parent events with correlated offsets and inherited target/task attributes, creating more session-reuse opportunities without revealing the regime label to the policy.Source: A. Experimental Setting and Online… · p. 7
  • References include JC-PPO, four deterministic heuristics, and Random Valid; SLA-Aware Greedy prioritizes the worst affected-request quality/freshness margin before incremental sensing cost.Source: A. Experimental Setting and Online… · p. 7

B. Training, Model Selection, and External Evaluation

  • Each learned method trains five seeds with the same physical-interaction budget, counting every replica trajectory used by common-trace methods.Source: B. Training, Model Selection, and… · p. 7
  • Checkpoint selection uses validation roots and a ranked return-and-constraint score without consulting external roots; selected checkpoints are then evaluated on external roots in both regimes.Source: B. Training, Model Selection, and… · p. 7

C. Component Ablation, Arrival-Load Stress, and Deployment Evaluation

CT-PPO leads the external operating-point comparison, while the supplied passages also specify the ablation, load-shift, deployment, and statistical evaluation protocols. The reported metrics distinguish return and service-resource outcomes from constraint excess.

C. Component Ablation, Arrival-Load Stress, and Deployment Evaluation

  • Adjacent contrasts test the factor-wise surrogate, common-trace reward credit, and prefix critics; Table I records final experimental configuration values from the effective artifacts.The ablation compares JC-PPO → Factorized-JC → CT-Reward → CT-PPO.Source: C. Component Ablation, Arrival-Load… · p. 7, Table I · p. 8
  • The frozen-checkpoint load-shift test varies only the independent per-tenant and clustered parent-event arrival rates across low, nominal, and high settings, reusing matched roots.It is a matched-root stress test, not an independent fresh-root replication across loads.Source: C. Component Ablation, Arrival-Load… · p. 7
  • Deployment evaluation compares reference and actor-only action-selection paths on CPU, checking action agreement on the first 100 observations per checkpoint pair before timing.Measurements use batch size one, one Torch thread, 100 warm-up observations, and alternating timing order.Source: C. Component Ablation, Arrival-Load… · p. 8

D. Metrics and Statistical Analysis

  • Episode return is the primary endpoint; constraint excess is reported as achieved excess, not strict feasibility, and paired effects use a hierarchical bootstrap over seeds and roots.The five training seeds are independent learned-policy units; roots and regimes are repeated matched workloads.Source: D. Metrics and Statistical Analysis · pp. 8–9

A. External Performance and Practical References

  • CT-PPO attains the highest mean return at 67.511, ahead of JC-PPO (66.576) and SLA-Aware Greedy (65.663).Table II reports the macro external operating points.Source: A. External Performance and Practical… · p. 9
  • Against SLA-Aware Greedy, CT-PPO gains 1.847 return, with +1.327 completed value, −2.602 sensing-resource cost, −0.348 positive constraint excess, and +0.0332 RPS.The reported 95% confidence interval for the return gain is [1.604, 2.094].Source: A. External Performance and Practical… · p. 9

B. Integrated CT-PPO–JC-PPO Effect and Consolidation Behavior

CT-PPO improves macro return over matched JC-PPO, with the gain located primarily in sensing-resource efficiency and greater consolidation. The ablation finds that common-trace reward credit drives the observed return improvement, while factorization alone and added prefix-critic machinery show no detectable incremental return gain in the evaluated configuration.

B. Integrated CT-PPO–JC-PPO Effect and Consolidation Behavior

  • CT-PPO improves macro return over JC-PPO by 0.934, reduces sensing cost by 6.277, and raises RPS by 0.0806.Completed value changes by −0.321; MERGE increases by 2.344 actions per episode and CREATE decreases by 2.272.Source: B. Integrated CT-PPO–JC-PPO Effect and… · p. 9
  • Under clustered arrivals, return improves by 1.042 and RPS by 0.1434, compared with 0.827 and 0.0177 under independent arrivals.Under clustered arrivals, sensing cost falls by 5.507 and positive constraint excess decreases by 0.665.Source: B. Integrated CT-PPO–JC-PPO Effect and… · p. 9
  • Static Compatibility Merge has higher RPS than CT-PPO (1.125 versus 1.114), but lower completed value (66.702 versus 74.005) and greater positive constraint excess (4.627 versus 2.164).The comparison illustrates why session reuse must be selective.Source: B. Integrated CT-PPO–JC-PPO Effect and… · p. 9

C. Component Attribution

  • The factor-wise surrogate alone yields no detectable return improvement: Factorized-JC minus JC-PPO is −0.099, with the interval spanning zero.Completed value decreases by 0.692 while sensing-resource cost decreases by 2.968.Source: C. Component Attribution · p. 9
  • After controlling for the surrogate change, common-trace reward credit raises return by 1.013; adding prefix critics changes return by only +0.021, with the interval spanning zero.The prefix-critic result is a non-detection, not an equivalence claim.Source: C. Component Attribution · p. 9

D. Arrival-Load Robustness

Across three tested loads, frozen CT-PPO checkpoints retain positive return effects over JC-PPO, while diagnostics show constraint-family tradeoffs and effectively unchanged actor-only CPU latency.

D. Arrival-Load Robustness

  • At every tested load, return effects remain positive in both arrival regimes, with larger gains under clustered arrivals.At the nominal rate, the independent and clustered effects are +0.827 and +1.042, respectively.Source: D. Arrival-Load Robustness · p. 10
  • At low load, sensing-resource cost falls 4.828 and RPS rises 0.0711, but completed value falls 0.275 and positive constraint excess rises 0.075.At high load, cost falls 7.521 and positive constraint excess falls 0.748, while completed value has no detectable change (−0.061 [−0.579, 0.428]); the consolidation shift remains more MERGE and fewer CREATE actions.Source: D. Arrival-Load Robustness · p. 10
  • CT–JC macro-return effects rise from +0.690 at rate 0.06 to +1.443 at 0.10, with all 15 load-by-training-seed contrasts positive.The gain also increases more under clustered arrivals, where reuse opportunities are richer; this trend is limited to the tested load grid.Source: D. Arrival-Load Robustness · p. 10, Table V · p. 11

E. Constraint and Training Diagnostics

  • CT-PPO’s macro E+ of 2.164 is entirely sensing-SLA excess; JC-PPO has 1.991 sensing-SLA excess and 0.392 communication-QoS excess.CT-PPO’s lower mean total excess reflects zero observed communication-QoS excess alongside a modestly higher sensing-SLA component.Source: E. Constraint and Training Diagnostics · p. 11
  • CT-PPO’s selected checkpoints occur at 790k–1000k slots and average latest-minus-best external return +0.037, while JC-PPO’s mean is −1.000.JC-PPO is selected earlier, at 20k–220k slots, and all five runs end below their selected checkpoints; the trajectories support validation-only checkpoint selection, not asymptotic convergence.Source: Figure 4 · p. 11, E. Constraint and Training Diagnostics · p. 11

F. Deployment Footprint and Inference Cost

VII. DISCUSSION AND LIMITATIONS

CT-PPO’s return gain is linked to selective session reuse and common-trace reward credit, with positive return differences across tested arrival loads. The discussion also qualifies this evidence by identifying constraint trade-offs, deployment details, and limits of the evaluated setting.

A. Mechanism and Component Attribution

  • Selective consolidation—not reuse for its own sake—shifts accepted demand toward existing sessions, and reuse is beneficial only when resource savings compensate for service loss.The factor-wise surrogate alone yields no detectable macro-return improvement; after controlling for it, common-trace reward credit produces positive return differences across five matched seeds, while prefix critics show no detectable incremental effect.Source: A. Mechanism and Component Attribution · p. 12

B. Workload Dependence and Constraint Trade-offs

  • CT-PPO retains a positive return advantage at all three tested arrival loads, but the service and constraint trade-offs vary with load.At low load, the gain is primarily a sensing-resource-efficiency effect accompanied by lower completed value and higher positive constraint excess; at high load, completed value shows no detectable difference while sensing cost and positive constraint excess decrease.Source: B. Workload Dependence and Constraint… · p. 12

C. Training and Deployment Implications

  • Common-trace credit requires replayable traces for same-workload policy replicas, while deployment uses one public observation and the trained actor; actor-only CPU latency is effectively unchanged relative to JC-PPO.Additional prefix-critic parameters are training-side; reported timing results are platform-specific and do not establish universally identical latency or lower computational complexity.Source: C. Training and Deployment Implications · p. 12

D. Scope and Limitations

  • Evidence is limited to a centralized, single-cell monostatic ISAC setting with fixed tenant, user, target, and task-generation settings and five training seeds.Different tenant or task mixes, mobility and channel statistics, multi-cell orchestration, and RF/hardware-in-the-loop or field validation remain untested.Source: D. Scope and Limitations · pp. 12–13

VIII. CONCLUSION

  • CT-PPO improves macro return over JC-PPO by 0.934 (95% CI [0.702, 1.164]) and over SLA-Aware Greedy by 1.847 across five seeds and matched external workloads.Relative to JC-PPO, it reduces sensing-resource cost by 6.277 and increases accepted requests per created session by 0.0806; the factor-wise surrogate alone yields no detectable macro-return improvement, while common-trace reward credit is the dominant empirical driver.Source: §VIII · p. 13

Related papers

Loading 2608.29256v1…