On a SONiC device using the PDDF platform framework (Celestica DS4000), intermittent PSU absence/recovery warnings are observed even though the PSUs are not physically removed.
The issue was observed for both PSU1 and PSU2.
show platform summary
Platform: x86_64-cel_silverstone_v2-r0
HwSKU: DS4000
ASIC: broadcom
ASIC Count: 1
Serial Number: R4023B2F044513GD200427
Model Number: R4023-F9003-A0
Hardware Revision: 4
Analysis:
- Initial analysis indicates that an intermittent BMC/IPMI command failure is being interpreted by the PDDF PSU presence logic as False (PSU not present).
- As a result, a transient failure to retrieve PSU presence information from the BMC is reported as a physical PSU removal. When the next BMC poll succeeds, the PSU is immediately reported as inserted again.
- Therefore, the current implementation does not distinguish between:
- PSU confirmed absent
- PSU confirmed present
- PSU presence status unavailable because the BMC/IPMI read failed
Observed Behavior
On August 29, 2026, multiple PSU1 absence/recovery events were observed.
Time Event Duration
10:07 PSU1 reported absent and then present ~6 seconds
13:46 PSU1 reported absent and then present ~30 seconds
14:46–14:48 PSU1 reported absent and then present ~1 min 20 sec
- Similar intermittent behavior has also been observed for PSU2.
- There was no corresponding physical PSU removal during these events.
- The warnings clear automatically once a subsequent BMC/IPMI read succeeds.
Code Flow
PSU presence is periodically polled by psud approximately every 2 seconds:
pmon / psud
|
| PSU2.get_presence()
v
pddf_psu.py::get_presence()
|
| get_attr_name_output("PSU2", "psu_present")
v
pddfapi.py
|
| pddf-device.json BMC command
| ipmitool raw 0x04 0x2D 0x3B
| index = 2
| mask = 0x01
v
raw_ipmi_get_request()
For a successful BMC response:
ipmitool raw 0x04 0x2D 0x3B
Example:
00 c0 01 80
^^
index 2
The configured mask is applied:
01 & 01 = 1
which produces:
status = "1"
The PDDF plugin contains a value map similar to:
{
"1": true,
"0": false
}
Therefore:
"1" -> True -> PSU present
"0" -> False -> PSU absent
However, when the BMC/IPMI command fails, times out, or does not return usable data, raw_ipmi_get_request() returns:
"N/A"
The resulting flow becomes:
BMC/IPMI read failure
|
v
raw_ipmi_get_request()
|
v
"N/A"
|
v
"N/A" not present in valmap
|
v
get_presence() returns False
|
v
psud interprets False as physical PSU absence
|
v
PSU absence warning
When the next poll succeeds:
BMC response OK
|
v
status = "1"
|
v
True
|
v
PSU reported inserted again
This produces misleading PSU removal/reinsertion events even though the hardware was never physically removed.
Actual Behavior
Currently:
ds4000_bmc_20260916T070339Z.zip
Valid BMC response, presence bit = 1
-> True
Valid BMC response, presence bit = 0
-> False
BMC timeout/error/invalid response
-> "N/A"
-> value not found in valmap
-> False
Therefore both of the following cases become indistinguishable to psud:
PSU genuinely absent -> False
BMC communication failure -> False
This can generate false PSU absence and recovery notifications.
Expected Behavior
A BMC/IPMI communication failure should not automatically be interpreted as:
PSU present = False
The software should distinguish between the following states:
Present
Absent
Unknown / read failure
Only an explicit, successfully decoded BMC response indicating that the PSU presence bit is 0 should result in a PSU-absent state.
A timeout, malformed response, command failure, exception, or "N/A" result should be treated as a sensor/read failure rather than confirmation that the PSU is physically absent.
On a SONiC device using the PDDF platform framework (Celestica DS4000), intermittent PSU absence/recovery warnings are observed even though the PSUs are not physically removed.
The issue was observed for both PSU1 and PSU2.
show platform summary
Analysis:
Observed Behavior
On August 29, 2026, multiple PSU1 absence/recovery events were observed.
Time Event Duration
10:07 PSU1 reported absent and then present ~6 seconds
13:46 PSU1 reported absent and then present ~30 seconds
14:46–14:48 PSU1 reported absent and then present ~1 min 20 sec
Code Flow
This produces misleading PSU removal/reinsertion events even though the hardware was never physically removed.
Actual Behavior
Currently:
ds4000_bmc_20260916T070339Z.zip
Therefore both of the following cases become indistinguishable to psud:
This can generate false PSU absence and recovery notifications.
Expected Behavior
A BMC/IPMI communication failure should not automatically be interpreted as:
PSU present = FalseThe software should distinguish between the following states:
Only an explicit, successfully decoded BMC response indicating that the PSU presence bit is 0 should result in a PSU-absent state.
A timeout, malformed response, command failure, exception, or "N/A" result should be treated as a sensor/read failure rather than confirmation that the PSU is physically absent.