You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
neuron-monitor: a period such as "1.0s" is silently replaced by the 5 s default #1419
neuron-monitor -c <config> accepts "period": "1s" but treats "1.0s", "0.5s" and "5.0s" as if no period were set and delivers the 5 s default. No error or warning is printed, and "500ms" produces no samples at all. The user guide's examples use "1s" and "2s" and say the default is 5 seconds, but do not say which formats are valid, so a period built from a float (f"{seconds}s") fails silently.
What we measured
Environment: inf2.xlarge, Deep Learning AMI Neuron (Ubuntu 22.04), aws-neuronx-tools 2.28.23.0 (kaena-tools/2.28), Neuron runtime 2.30.51, 2026-09-11. Each sample's own period field and the number of samples in 16 s:
"period" in the config
period reported by the samples
samples in 16 s
"1.0s"
5.0 s
3
"1s"
1.0 s
13
"0.5s"
5.0 s
"5.0s"
5.0 s
"500ms"
no samples at all
0
Config shape (system metrics group omitted for brevity):
Run with neuron-monitor -c config.json while a Neuron workload is running, and read period from each report.
Why it matters
A tool that builds the period from a float gets 5 s samples while believing it asked for 1 s, and nothing reports the gap. We hit this ourselves and misattributed it to the tool ("neuron-monitor floors at ~2 s") before comparing the requested period with the delivered one.
Suggested change
Either accept "1.0s" and similar, or fail with a message that names the accepted format; and state the accepted format (whole seconds, Ns) and the behaviour for an unparsed value in the neuron-monitor user guide.
Not checked: tools releases newer than 2.28.23.0. If the behaviour has changed, a one-line note in the release notes would be enough for us.
Our write-up with the code that stopped sending the bad string: https://github.com/pantheongpu/pantheonneuron (docs/checks_that_pass_by_accident.md, "neuron-monitor accepts whole seconds only").
Activity
github-actions commentedon Oct 9, 2026
Hi @saqibkh, Thank you for filing the issue! We will take a look and get back to you.