Skip to content

[chassisd] Immediately restart DPU module on kernel panic without waiting for grace period - #852

Open
yush-1018 wants to merge 1 commit into
sonic-net:masterfrom
yush-1018:mtfuji-dpu-panic-restart-fix
Open

yush-1018 wants to merge 1 commit into
sonic-net:masterfrom
yush-1018:mtfuji-dpu-panic-restart-fix

Conversation

@yush-1018

Copy link
Copy Markdown

Description

Currently, when a DPU module encounters a kernel panic and restarts, chassisd transitions to the DPU_STATE_WAIT_FOR_SELF_RECOVERY state and waits for a full 300-second (5-minute) grace period to elapse before performing a recovery power cycle. Since a kernel panic is an unrecoverable state, waiting for this timeout causes unnecessary downtime.

This PR modifies the state machine in chassisd to check the reboot cause. If the cause matches "kernel panic", it immediately bypasses the 300-second wait time and directly triggers a recovery power-cycle to restore services as fast as possible.

Changes Made

  • Auto-Recovery Optimization:
    • Updated update_dpu_recovery_state() in chassisd to query get_reboot_cause(). If the cause or its details contain "kernel panic", the 300-second self-recovery timeout is bypassed, and the state machine directly enters DPU_STATE_POWER_CYCLE (or ManualIntervention if auto-recovery is disabled).
  • Unit Tests & Windows Compatability:
    • Added test_self_recovery_kernel_panic_bypasses_grace_period() in tests/test_dpu_auto_recovery.py to verify that a mocked kernel panic reboot cause bypasses the waiting period.
    • Added a fallback from os.symlink to shutil.copy in persist_dpu_reboot_cause() to prevent Windows administrative privilege errors during testing.
    • Mocked syslog and swsscommon dependencies in sonic-chassisd/tests/conftest.py to support unit test verification on Windows environments.

Verification

  • Executed the chassis auto-recovery tests locally: All 140/140 unit tests are fully passing.
============================= 140 passed in 2.54s =============================

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Not Signed

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

---Powered by SONiC BuildBot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants