Skip to content

neuron-monitor: a period such as "1.0s" is silently replaced by the 5 s default #1419

Open

Description

@saqibkh

Summary

neuron-monitor -c <config> accepts "period": "1s" but treats "1.0s", "0.5s" and "5.0s" as if no period were set and delivers the 5 s default. No error or warning is printed, and "500ms" produces no samples at all. The user guide's examples use "1s" and "2s" and say the default is 5 seconds, but do not say which formats are valid, so a period built from a float (f"{seconds}s") fails silently.

What we measured

Environment: inf2.xlarge, Deep Learning AMI Neuron (Ubuntu 22.04), aws-neuronx-tools 2.28.23.0 (kaena-tools/2.28), Neuron runtime 2.30.51, 2026-09-11. Each sample's own period field and the number of samples in 16 s:

"period" in the config period reported by the samples samples in 16 s
"1.0s" 5.0 s 3
"1s" 1.0 s 13
"0.5s" 5.0 s
"5.0s" 5.0 s
"500ms" no samples at all 0

Config shape (system metrics group omitted for brevity):

{"period": "1.0s",
 "neuron_runtimes": [{"tag_filter": ".*", "metrics": [{"type": "neuroncore_counters"}]}],
 "system_metrics": [{"type": "neuron_hw_counters"}]}

Run with neuron-monitor -c config.json while a Neuron workload is running, and read period from each report.

Why it matters

A tool that builds the period from a float gets 5 s samples while believing it asked for 1 s, and nothing reports the gap. We hit this ourselves and misattributed it to the tool ("neuron-monitor floors at ~2 s") before comparing the requested period with the delivered one.

Suggested change

Either accept "1.0s" and similar, or fail with a message that names the accepted format; and state the accepted format (whole seconds, Ns) and the behaviour for an unparsed value in the neuron-monitor user guide.

Not checked: tools releases newer than 2.28.23.0. If the behaviour has changed, a one-line note in the release notes would be enough for us.

Our write-up with the code that stopped sending the bad string: https://github.com/pantheongpu/pantheonneuron (docs/checks_that_pass_by_accident.md, "neuron-monitor accepts whole seconds only").

Activity

  1. github-actions commented on Oct 9, 2026

    @github-actions

    Hi @saqibkh, Thank you for filing the issue! We will take a look and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions