Reference simulator and reproducible numerical results for the paper
Lyapunov-Guided Safe Reinforcement Learning and Risk-Budget Feasibility for Energy-Aware O-RAN Scheduling Liang Dong, Senior Member, IEEE. Under review, IEEE Transactions on Cognitive Communications and Networking.
The codebase implements a primal-dual actor-critic for a K-cell O-RAN cluster with inter-cell offloading. A PyTorch policy is constrained by a CVaR budget on the per-slot loss (Rockafellar-Uryasev form, PID multiplier with anti-windup), and every action passes through a Lyapunov-guided safety filter: a per-cell lower-confidence-bound projection with a joint fallback that wakes a backlogged cell and the neighbors that would offload onto it. Separate critics are trained on the energy and risk channels.
The R2 code and results replace those of the first revision. An audit of the analysis and code found errors that affected several published claims; all were fixed and every experiment was rerun. In brief:
- Learned baselines. PPO-Lagrangian's multiplier used the mean loss while
its critic and actor used the CVaR surrogate; CRPO fed a routing multiplier
of 1e6 to the networks as an input (2e4 after scaling), saturating every
unit; WCSAC fitted one critic to two incompatible targets. All three are
corrected (
sim/safe_baselines.py), and their budgets are calibrated on validation seeds at the same tightness relative to always-on. WCSAC-GS follows the source's structure: state-action mean and variance critics on its Bellman targets (Eqs. 14-15, fitted by squared error) and its actor step against the fixed safety measure Q_c + k sqrt(V_c), taken by the likelihood-ratio estimator inside PPO.sim/check_wcsac.pychecks the targets and the sign of the actor step on an exact two-step MDP. Its critics use lambda-return forms of the source's targets (the variance in its equivalent direct form) with a target-network step after every gradient step, and its budget is calibrated by Monte Carlo regression under always-on (sim/r2.py, _wcsac_ao_stat_mc). - Textbook drift-plus-penalty.
TextbookDPPminimizes V P + sum_b Q_b (E[A_b | a] - E[mu_b | a]) jointly over the sleep patterns the hysteresis admits and the share endpoints (0 for a cell locked awake by its minimum dwell, 0.05 otherwise, and 1), with offloading anticipated through the offload map (sim/baselines.py);sim/check_dpp.pychecks it exhaustively against the environment's feasible actions. - Safety filter. It now predicts arrivals under the sleep pattern that will
execute, overrides the hysteresis dwell for safety wakes, charges the wake-up
service loss, and executes a joint fallback when its projection is
infeasible, logging each case (
sim/safety_filter.py). The earlier fallback raised only the backlogged cell's own share, which an all-sleep adversary defeats. - Other fixes. Phase feature aligned with the compressed daily cycle; CVaR threshold step ordered after the risk cost; PID anti-windup; exact empirical CVaR at ties; correlated channel that preserves the Lumos5G marginal up to a finite-grid rank transform; hexagonal topology for K = 19, 37, 61; loaders that refuse synthetic substitutes; the rule formerly called drift-plus-penalty renamed backlog-proportional, with a textbook drift-plus-penalty controller added.
- Protocol. Tuning and budget calibration use ten validation seeds; every reported number uses ten separate test seeds. Each learned controller is trained once per seed (fresh arrival realization per training episode), its checkpoint is saved, and all evaluations load it, so comparisons are paired.
Ten test seeds, 100 s evaluation episodes at 10 ms slots, seven cells,
beta = 0.95, Gamma = 3.5, 3200 training updates, 95% bootstrap intervals
over seeds:
| Controller | Power (W) | CVaR_0.95 | P{loss > Gamma} | Toggles/min |
|---|---|---|---|---|
| Always-on reference | 1610.0 | 3.050 | 0.7% | 0 |
| Backlog-proportional, rho = 0.875 (validation-selected) | 1151.0 | 3.408 | 1.2% | 0 |
| Textbook drift-plus-penalty, V = 4e-4 (validation-selected) | 1373.3 | 3.252 | 1.0% | 1375 |
| PPO-Lagrangian + LCB filter | 1393.7 | 3.320 | 1.1% | 5 |
| CRPO + LCB filter | 1218.5 | 3.717 | 2.3% | 1022 |
| WCSAC-GS + LCB filter | 1443.0 | 3.727 | 3.4% | 644 |
| Proposed | 1343.6 | 3.468 | 1.5% | 53 |
The risk budget must be screened. The loss contains an arrival term no
scheduler influences. A certified floor Gamma_0 = 2.019 (valid for unequally
loaded cells when eta * max(Abar) <= min(Abar), which holds on every trace)
rules out smaller budgets, and the always-on reference Gamma_AO = 3.050 is a
feasible witness for larger ones. Budgets in between are undetermined.
A simple rule is not beaten. Among filtered learned methods whose mean tail meets the budget, the proposed controller draws the least power, but the backlog-proportional rule draws 192.7 W less (paired interval [162, 227]) with no statistically detected difference in tail (+0.06, [-0.04, 0.18]). The textbook drift-plus-penalty controller meets the budget at 1373.3 W with a tighter tail than the proposed controller, but toggles 1375 times per minute. CVaR values are mean empirical CVaR over the ten test seeds; the proposed controller's interval [3.35, 3.61] crosses the budget.
The filter carries the tail. Evaluating the trained policy with the filter bypassed raises CVaR from 3.468 to 7.280; the filter costs 23 W ([2, 50]) and acts on 4.8% of cell-slots, falling back to full service in 94% of those. With risk pressure removed, it holds the seed-averaged time-average backlog below 13 Mb against an adversary that asks every cell to sleep, versus up to 14,992 Mb without it. This is empirical evidence: the queue theorem requires the true conditional drift condition, which the filter checks only with estimates.
The paper does not claim an O(1/V) energy-gap guarantee: a per-state safety filter can carry an energy price that no Lyapunov weight removes, and the earlier version's claim to the contrary was incorrect.
sim/
├── config.py Calibrated parameters (single source of truth)
├── arrivals.py Per-cell arrival profiles from Shanghai Telecom
├── channel_lumos5g.py Per-slot service-rate multiplier from Lumos5G
├── env.py K-cell environment: queues, energy, offloading, channel
├── networks.py Factored actor (sleep x share) and critics
├── safety_filter.py LCB projection with joint fallback; aggregate comparison
├── algorithm.py Primal-dual PPO, two critics, PID dual, CVaR threshold
├── baselines.py Classical controllers and the shared evaluation routine
├── safe_baselines.py PPO-Lagrangian, CRPO, WCSAC-GS on the same backbone
├── metrics.py Exact empirical CVaR, bootstrap intervals
├── r2.py R2 experiment suite (calibrate ... timing)
├── r2_scale_classical.py Classical references on the larger clusters
├── make_r2_figures.py TikZ/pgfplots figures from sim/results/r2
├── make_r2_tables.py LaTeX table and paired statistics
├── make_tikz_figures.py Shared figure preamble (and the R1 figure set)
└── revision.py, ... R1 drivers, kept for the record of the R1 results
fig/ Figures (PDF) and their TikZ sources
tab/ Generated LaTeX tables
sim/results/r2/ R2 results: JSON summaries, checkpoints, per-slot series
sim/results/ R1 results (superseded, kept for the record)
sim/data/ Datasets (gitignored, see below)
Two real datasets drive the experiments. Neither is committed; reacquire them
from the public sources below and place them under sim/data/.
| Dataset | Used for | Source |
|---|---|---|
| Shanghai Telecom (Wang et al., IEEE TMC 2021) | Per-cell hourly load profile (seven busiest cells, June 1-15, 2014) | https://wangshangguang.github.io/telecom_dataset/ |
| Lumos5G (Narayanan et al., ACM IMC 2020) | Per-slot mmWave service-rate multiplier | https://github.com/SIGCOMM21-Lumos5G/lumos5g |
config.require_real_data defaults to True: both loaders raise rather than
substitute a synthetic profile or channel pool, including from a cached file,
and every result file records its data sources and a hash of the simulator
code under _provenance.
Arrivals are trace-shaped, not recorded slot by slot. Each episode compresses
one 24-hour profile onto its slots; the nominal load is a scaled Poisson draw
at 0.3 * h_b(t) / max(h_b) Mb per slot (0.3 Mb is the busy-hour rate) plus
bursts (1 + Y) * 1.5 * m with probability 0.04, where Y is Lomax (Pareto II)
with shape 2.5 (NumPy's pareto). Realized cell means are 0.18-0.25 Mb per
slot. The Lumos5G multiplier is drawn from all 68,118 one-second throughput
samples of the public release (23% of them logged on LTE), normalized by the
median (424 Mbit/s) and clipped to [0.1, 2.0]; the clip binds on 26.1% of
samples above and 10.0% below.
Requires Python 3.10+, PyTorch 2.x, NumPy, SciPy, pandas, openpyxl, and a LaTeX installation with pgfplots for the figures. The reported runs used Python 3.12, PyTorch 2.11 and NumPy 2.2 on an Intel Core i7-12700KF.
# Full R2 campaign (about 3.5 hours on 20 CPU workers)
./sim/results/r2/run_all.sh
# or stage by stage
OMP_NUM_THREADS=1 python3 -m sim.r2 calibrate
OMP_NUM_THREADS=1 python3 -m sim.r2 feasible
OMP_NUM_THREADS=1 python3 -m sim.r2 classical
OMP_NUM_THREADS=1 python3 -m sim.r2 train # 250 training runs
OMP_NUM_THREADS=1 python3 -m sim.r2 evaluate
OMP_NUM_THREADS=1 python3 -m sim.r2 stress
OMP_NUM_THREADS=1 python3 -m sim.r2 corr
OMP_NUM_THREADS=1 python3 -m sim.r2 timing
OMP_NUM_THREADS=1 python3 -m sim.r2_scale_classical
OMP_NUM_THREADS=1 python3 -m sim.r2_single_fallback # single-cell fallback ablation
python3 -m sim.check_wcsac # exact WCSAC estimator check
python3 -m sim.check_dpp # exhaustive DPP optimality check
# The released WCSAC-GS and textbook DPP results were produced after the main
# campaign by the targeted rerun below: Monte Carlo calibration of the WCSAC
# budget on validation seeds (with a held-out accuracy check and the TD-critic
# diagnostic), the DPP validation grid and selection, test evaluation of the
# DPP grid, and retraining/evaluation of WCSAC and WCSAC+F. The log is
# sim/results/r2/rerun_audit3.log; replaced entries are kept under "_r2draft"
# keys, superseded checkpoints under ckpt/_superseded/ and superseded per-slot
# series under series/_superseded/. Every result JSON records the provenance
# of each stage that wrote it (_provenance, _provenance_history).
OMP_NUM_THREADS=1 python3 -m sim.r2_rerun all
# Figures, table and paired statistics from sim/results/r2
python3 -m sim.make_r2_figures
python3 -m sim.make_r2_tablesSet OMP_NUM_THREADS=1: the networks are small enough that intra-op
threading costs more than it saves, and single-threaded reductions make a
rerun reproducible. The training stage skips any configuration whose final
checkpoint already exists, so the released checkpoints let evaluate and the
later stages run without retraining; delete a configuration's checkpoints to
retrain it. Saved per-slot series (sim/results/r2/series/) hold loss, power
and total backlog for the main evaluations, so CVaR, power and backlog
summaries can be recomputed from them; delay-proxy, toggle, share and filter
statistics, and the stress, sweep and scaling summaries, need a replay. Test seeds are the first ten outputs of
numpy.random.SeedSequence(20260601) and validation seeds the next ten;
evaluation traces use seed + 9000.
MIT. See LICENSE.
@article{Dong_SafeRL_TCCN_2026,
author = {Liang Dong},
title = {Lyapunov-Guided Safe Reinforcement Learning and Risk-Budget
Feasibility for Energy-Aware {O-RAN} Scheduling},
journal = {IEEE Transactions on Cognitive Communications and Networking},
year = {2026},
note = {Under review.}
}