Skip to content

Delete Volume when creation fails with status ERROR - #1452

Open
nschad wants to merge 9 commits into
mainfrom
delete-vol-when-error-create
Open

nschad wants to merge 9 commits into
mainfrom
delete-vol-when-error-create

Conversation

@nschad

@nschad nschad commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

How to categorize this PR?

/kind enhancement

What this PR does / why we need it:

The idea is to automatically delete the volume when status is ERROR. We do this there is no left-over volumes even though the CreateVolume RPC failed.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Breaking changes:

Signed-off-by: Niclas Schad <niclas.schad@stackit.cloud>
@ske-prow

ske-prow Bot commented Aug 4, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ske-prow ske-prow Bot added kind/enhancement Enhancement, improvement, extension do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Aug 4, 2026
@ske-prow

ske-prow Bot commented Aug 4, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nschad for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ske-prow ske-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 4, 2026
Comment thread pkg/csi/blockstorage/controllerserver.go Outdated
Comment thread pkg/csi/blockstorage/controllerserver.go
Comment thread pkg/csi/blockstorage/controllerserver.go Outdated
nschad added 3 commits August 4, 2026 13:33
Signed-off-by: Niclas Schad <niclas.schad@stackit.cloud>
Signed-off-by: Niclas Schad <niclas.schad@stackit.cloud>
Signed-off-by: Niclas Schad <niclas.schad@stackit.cloud>
@ske-prow ske-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 5, 2026
Comment thread pkg/csi/blockstorage/controllerserver.go Outdated
Comment thread pkg/csi/blockstorage/controllerserver.go Outdated
Signed-off-by: Niclas Schad <niclas.schad@stackit.cloud>

@stackit-ske-bot stackit-ske-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKE Code Review

Architectural Feedback

  • Simplify WaitVolumeTargetStatusWithCustomBackoff Signature:
    In pkg/stackit/client/iaas.go#L52, WaitVolumeTargetStatusWithCustomBackoff was changed from accepting volumeID string to vol **iaas.Volume to allow mutating the pointer in-place for cleanup after creation. However, since the subsequent cleanup call after WaitVolumeTargetStatusWithCustomBackoff was removed in commit 87917aa, mutating vol in-place is no longer needed in pkg/csi/blockstorage/controllerserver.go#L271.

    Passing a double pointer (**iaas.Volume) across interface boundaries is unidiomatic in Go, introduces nil dereference risks (e.g. (*vol).GetId()), and forced modifications across multiple unit tests in pkg/csi/blockstorage/controllerserver_test.go and pkg/stackit/client/mock/iaas_mock.go. Reverting the interface method back to volumeID string simplifies the API, avoids unnecessary mock changes, and keeps it consistent with WaitVolumeTargetStatus.

Findings & Feedback

  • All findings and concrete recommendations have been provided as inline code suggestions above.

Verdict

Comment — The flag and cleanup logic for existing error-state volumes in CreateVolume are a great improvement. Please address the nil-safety items and consider reverting the double-pointer signature in WaitVolumeTargetStatusWithCustomBackoff.

Comment thread pkg/csi/blockstorage/controllerserver.go Outdated
Comment thread pkg/csi/blockstorage/controllerserver.go
Comment thread pkg/csi/blockstorage/controllerserver.go
Comment thread pkg/stackit/client/iaas.go Outdated
@breuerfelix
breuerfelix marked this pull request as ready for review September 15, 2026 14:31
@nschad nschad changed the title WIP: Delete Volume when creation fails with status ERROR Delete Volume when creation fails with status ERROR Sep 15, 2026
@ske-prow ske-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 15, 2026

@stackit-ske-bot stackit-ske-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKE Code Review

Architectural Feedback

  • Handling of All Provisioning Error States in Cleanup:
    In pkg/csi/blockstorage/controllerserver.go#L298, deleteVolumeInError strictly compares against stackitclient.VolumeErrorStatus ("ERROR"):

    if vol.GetStatus() != stackitclient.VolumeErrorStatus {
        return
    }

    However, as defined in pkg/stackit/client/iaas.go#L98 (volumeErrorStates), STACKIT IaaS volume provisioning can enter several terminal error states—most notably "ERROR_KMS-ENCRYPTION-PARAMS" if encrypted volume creation fails (e.g. invalid KMS keys or service account permissions), as well as "ERROR_RESTORING-BACKUP".

    If an encrypted volume fails creation and enters "ERROR_KMS-ENCRYPTION-PARAMS", subsequent retries by Kubernetes external-provisioner will find len(vols) == 1, but deleteVolumeInError will skip deletion because the status does not match "ERROR". The volume is then left orphaned and the provisioning loop remains permanently stuck returning codes.Internal.

    Recommendation: Expose an error-state classification function in pkg/stackit/client (e.g., IsVolumeErrorState(status string) bool matching creation error states and excluding non-terminal or deleting states like ERROR_DELETING), and use it within deleteVolumeInError to ensure all failed provisioning artifacts are cleaned up when deleteVolumesInErrorState is enabled.

Findings & Feedback

  • All findings and concrete recommendations have been provided as inline code suggestions above.

Verdict

Comment — The refactored WaitVolumeTargetStatusWithCustomBackoff method returning (*iaas.Volume, error) and the adoption of getter methods across the controller look very clean. Please consider extending the error status check in deleteVolumeInError to handle all provisioning error states (such as KMS errors) as outlined in the architectural feedback.

Comment thread pkg/stackit/client/iaas.go Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement Enhancement, improvement, extension size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants