Skip to content

Bug:[PDDF][PSU] BMC/IPMI read failure incorrectly reported as PSU absent due to get_presence() mapping "N/A" to False #901

Description

@ssithaia-ebay

On a SONiC device using the PDDF platform framework (Celestica DS4000), intermittent PSU absence/recovery warnings are observed even though the PSUs are not physically removed.
The issue was observed for both PSU1 and PSU2.

show platform summary

Platform: x86_64-cel_silverstone_v2-r0
HwSKU: DS4000
ASIC: broadcom
ASIC Count: 1
Serial Number: R4023B2F044513GD200427
Model Number: R4023-F9003-A0
Hardware Revision: 4

Analysis:

  • Initial analysis indicates that an intermittent BMC/IPMI command failure is being interpreted by the PDDF PSU presence logic as False (PSU not present).
  • As a result, a transient failure to retrieve PSU presence information from the BMC is reported as a physical PSU removal. When the next BMC poll succeeds, the PSU is immediately reported as inserted again.
  • Therefore, the current implementation does not distinguish between:
    • PSU confirmed absent
    • PSU confirmed present
    • PSU presence status unavailable because the BMC/IPMI read failed

Observed Behavior
On August 29, 2026, multiple PSU1 absence/recovery events were observed.
Time Event Duration
10:07 PSU1 reported absent and then present ~6 seconds
13:46 PSU1 reported absent and then present ~30 seconds
14:46–14:48 PSU1 reported absent and then present ~1 min 20 sec

  • Similar intermittent behavior has also been observed for PSU2.
  • There was no corresponding physical PSU removal during these events.
  • The warnings clear automatically once a subsequent BMC/IPMI read succeeds.

Code Flow

PSU presence is periodically polled by psud approximately every 2 seconds:
pmon / psud
|
| PSU2.get_presence()
v
pddf_psu.py::get_presence()
|
| get_attr_name_output("PSU2", "psu_present")
v
pddfapi.py
|
| pddf-device.json BMC command
| ipmitool raw 0x04 0x2D 0x3B
| index = 2
| mask = 0x01
v
raw_ipmi_get_request()
For a successful BMC response:
ipmitool raw 0x04 0x2D 0x3B

Example:
00 c0 01 80
^^
index 2
The configured mask is applied:
01 & 01 = 1
which produces:
status = "1"
The PDDF plugin contains a value map similar to:
{
"1": true,
"0": false
}
Therefore:
"1" -> True -> PSU present
"0" -> False -> PSU absent
However, when the BMC/IPMI command fails, times out, or does not return usable data, raw_ipmi_get_request() returns:
"N/A"
The resulting flow becomes:
BMC/IPMI read failure
|
v
raw_ipmi_get_request()
|
v
"N/A"
|
v
"N/A" not present in valmap
|
v
get_presence() returns False
|
v
psud interprets False as physical PSU absence
|
v
PSU absence warning
When the next poll succeeds:
BMC response OK
|
v
status = "1"
|
v
True
|
v
PSU reported inserted again

This produces misleading PSU removal/reinsertion events even though the hardware was never physically removed.

Actual Behavior
Currently:

ds4000_bmc_20260916T070339Z.zip

Valid BMC response, presence bit = 1
-> True
Valid BMC response, presence bit = 0
-> False
BMC timeout/error/invalid response
-> "N/A"
-> value not found in valmap
-> False

Therefore both of the following cases become indistinguishable to psud:

PSU genuinely absent -> False
BMC communication failure -> False

This can generate false PSU absence and recovery notifications.

Expected Behavior
A BMC/IPMI communication failure should not automatically be interpreted as:
PSU present = False
The software should distinguish between the following states:

Present
Absent
Unknown / read failure

Only an explicit, successfully decoded BMC response indicating that the PSU presence bit is 0 should result in a PSU-absent state.
A timeout, malformed response, command failure, exception, or "N/A" result should be treated as a sensor/read failure rather than confirmation that the PSU is physically absent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions