Repository navigation
feat: show the encryption state of artifacts and add routes to encrypt the files stored before - #11561
feat: show the encryption state of artifacts and add routes to encrypt the files stored before#11561DharmaBytesX wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
2 issues found across 40 files
Confidence score: 4/5
- The generated SDK protocols in
protocols.pystill omitCoreArtifact.encryption, so SDK users may not see the field added to GraphQL. Update the protocols to include it. get-artifact-encryption-notice.tsaccepts arbitrary strings forUNREADABLE_REASONS, so a new backend reason could fall back to a generic message silently. Type the mapping against the API enum so additions are checked.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/infrahub/core/schema/definitions/core/artifact.py">
<violation number="1" location="backend/infrahub/core/schema/definitions/core/artifact.py:75">
P2: `CoreArtifact.encryption` is now part of GraphQL, but the generated SDK protocols still omit it at opsmill/infrahub-sdk-python/infrahub_sdk/protocols.py:370-380 and opsmill/infrahub-sdk-python/infrahub_sdk/protocols.py:1513-1523, so typed SDK consumers cannot read the new value. Regenerate the SDK protocols before shipping this API change.</violation>
</file>
<file name="frontend/app/src/entities/artifacts/domain/rules/get-artifact-encryption-notice.ts">
<violation number="1" location="frontend/app/src/entities/artifacts/domain/rules/get-artifact-encryption-notice.ts:8">
P2: `UNREADABLE_REASONS` accepts arbitrary strings even though `UnreadableReason` is a closed API enum, so adding a backend reason will silently produce a generic message. Type the mapping against the enum (excluding the separately handled `missing` value) and add an exhaustiveness check so new reasons require corresponding UI copy.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Re-trigger cubic
| | { level: "warning"; message: string } | ||
| | { level: "error"; message: string }; | ||
|
|
||
| const UNREADABLE_REASONS: Record<string, string> = { |
There was a problem hiding this comment.
P2: UNREADABLE_REASONS accepts arbitrary strings even though UnreadableReason is a closed API enum, so adding a backend reason will silently produce a generic message. Type the mapping against the enum (excluding the separately handled missing value) and add an exhaustiveness check so new reasons require corresponding UI copy.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At frontend/app/src/entities/artifacts/domain/rules/get-artifact-encryption-notice.ts, line 8:
<comment>`UNREADABLE_REASONS` accepts arbitrary strings even though `UnreadableReason` is a closed API enum, so adding a backend reason will silently produce a generic message. Type the mapping against the enum (excluding the separately handled `missing` value) and add an exhaustiveness check so new reasons require corresponding UI copy.</comment>
<file context>
@@ -0,0 +1,59 @@
+ | { level: "warning"; message: string }
+ | { level: "error"; message: string };
+
+const UNREADABLE_REASONS: Record<string, string> = {
+ unknown_key: "It was not encrypted with the current key.",
+ no_key: "No encryption key is configured.",
</file context>
There was a problem hiding this comment.
Fixed in 4781b77: the map is keyed by the API reasons without missing. A new reason does not compile until it has a text.
| branchName: currentBranch.name, | ||
| }); | ||
| if (result.updated) { | ||
| await queryClient.invalidateQueries({ queryKey: objectQueryKeys.all }); |
There was a problem hiding this comment.
we probably should also invalidate artifactsQueryKeys.file
There was a problem hiding this comment.
Not changed: the check updates only encryption. artifactsQueryKeys.file is keyed by storage_id, which the check does not change, and the file query reads /api/storage/object/{storage_id}. If you know a case where the content changes, tell me and I add it.
There was a problem hiding this comment.
All reported issues were addressed across 14 files (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Confidence score: 2/5
check-artifact-encryption.query.tslets signed-in callers without CoreArtifact view permission probe known IDs for encryption or key status. Enforce the artifact view permission before returning the check result.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="frontend/app/src/entities/artifacts/ui/queries/check-artifact-encryption.query.ts">
<violation number="1" location="frontend/app/src/entities/artifacts/ui/queries/check-artifact-encryption.query.ts:35">
P1: Authentication here does not authorize the encryption check: signed-in callers without CoreArtifact view permission can probe known IDs and learn the file's encryption or key status. Enforce the artifact view permission in the backend endpoint before loading or inspecting the file; the frontend `enabled` condition cannot provide authorization.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Re-trigger cubic
| } | ||
| return result; | ||
| }, | ||
| enabled: isAuthenticated && timeMachineDate === null, |
There was a problem hiding this comment.
P1: Authentication here does not authorize the encryption check: signed-in callers without CoreArtifact view permission can probe known IDs and learn the file's encryption or key status. Enforce the artifact view permission in the backend endpoint before loading or inspecting the file; the frontend enabled condition cannot provide authorization.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At frontend/app/src/entities/artifacts/ui/queries/check-artifact-encryption.query.ts, line 35:
<comment>Authentication here does not authorize the encryption check: signed-in callers without CoreArtifact view permission can probe known IDs and learn the file's encryption or key status. Enforce the artifact view permission in the backend endpoint before loading or inspecting the file; the frontend `enabled` condition cannot provide authorization.</comment>
<file context>
@@ -30,7 +32,7 @@ export const useCheckArtifactEncryption = (artifactId: string) => {
return result;
},
- enabled: timeMachineDate === null,
+ enabled: isAuthenticated && timeMachineDate === null,
staleTime: 0,
refetchOnWindowFocus: false,
</file context>
There was a problem hiding this comment.
Not changed in this PR. This check follows the existing content endpoint, GET /api/artifact/{id}, which returns the whole file to any signed-in user without a view permission check. The check reveals less: one of three values and a reason, for an ID the caller must already know. This is listed as a decision for reviewers in the description. A view permission check can be added to both endpoints together in a later change.
There was a problem hiding this comment.
7 issues found across 24 files (changes from recent commits).
Confidence score: 2/5
- In
encryption_migration.py, the shielded migration task can outlive the request-scopeddbsession, so a client disconnect can make the migration fail before checksum lookup. Keep the session alive for the task or give the task its own session. - In
storage.py, an unconditional S3 PUT can recreate an object deleted after the migration reads it, while reporting success. Use a conditional replacement and report when the object is missing or has changed. - In
storage.py, the local size check and read can target different files if the path is atomically replaced, letting a larger file exceed the memory budget. Open the file once and check and read through that handle. - In
migrate.py, a failed confirming header read is reported as though the encrypted copy could not be replaced, even if the write already succeeded. Handle inspection errors separately from replacement errors.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="changelog/+encryption-existing-files.added.md">
<violation number="1" location="changelog/+encryption-existing-files.added.md:1">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
This PR adds the server routes, not the `infrahubctl storage encrypt` command or its progress and resume behavior; those are in the separate SDK change. Limit this fragment to the server-side functionality added here.</violation>
<violation number="2" location="changelog/+encryption-existing-files.added.md:1">
P3: The release note says there are settings that limit the pace, but no setting does that. The two settings limit the memory budget and the largest file size, and the client sets the pace. Reword to name the settings that exist.</violation>
<violation number="3" location="changelog/+encryption-existing-files.added.md:1">
P3: This release note overstates the checksumless-file behavior: `--allow-unverified` intentionally encrypts those files without a recorded checksum. Say they are left unchanged by default and document the opt-in exception.</violation>
</file>
<file name="backend/infrahub/storage.py">
<violation number="1" location="backend/infrahub/storage.py:208">
P2: This unconditional S3 PUT can recreate an object deleted after migration reads it, so the endpoint reports success and resurrects deleted storage. Use an ETag/version-conditional replacement and report a missing or conflict result when it no longer exists.</violation>
<violation number="2" location="backend/infrahub/storage.py:471">
P2: The local size check and read use separate path operations, so an atomic replacement can swap in a larger object after `stat` and before `read_bytes`, exceeding the memory budget. Open the file first and fstat/read that descriptor.</violation>
</file>
<file name="backend/infrahub/storage_encryption/migrate.py">
<violation number="1" location="backend/infrahub/storage_encryption/migrate.py:170">
P2: The reason "the object could not be replaced" also covers a failure of the confirming header read, after the encrypted copy was already written. Move the `inspect_encryption` call out of this try block, or give it its own message, so the result does not claim a failed write.</violation>
</file>
<file name="backend/infrahub/api/storage/encryption_migration.py">
<violation number="1" location="backend/infrahub/api/storage/encryption_migration.py:143">
P1: The shielded migration task outlives the request-scoped `db` session it uses. A client disconnect can close that session before the task reaches `get_recorded_checksums`, causing the migration to fail instead of completing the promised post-cancellation write. Run the background task with its own database session, or keep the request-scoped session alive for the task's full lifetime.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack
View guided diff | Re-trigger cubic
| HTTPException: With status 409 when storage encryption is disabled or this process has no storage key. | ||
|
|
||
| """ | ||
| task = asyncio.create_task( |
There was a problem hiding this comment.
P1: The shielded migration task outlives the request-scoped db session it uses. A client disconnect can close that session before the task reaches get_recorded_checksums, causing the migration to fail instead of completing the promised post-cancellation write. Run the background task with its own database session, or keep the request-scoped session alive for the task's full lifetime.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/api/storage/encryption_migration.py, line 143:
<comment>The shielded migration task outlives the request-scoped `db` session it uses. A client disconnect can close that session before the task reaches `get_recorded_checksums`, causing the migration to fail instead of completing the promised post-cancellation write. Run the background task with its own database session, or keep the request-scoped session alive for the task's full lifetime.</comment>
<file context>
@@ -0,0 +1,225 @@
+ HTTPException: With status 409 when storage encryption is disabled or this process has no storage key.
+
+ """
+ task = asyncio.create_task(
+ encrypt_stored_objects(
+ db=db,
</file context>
| @@ -0,0 +1 @@ | |||
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |||
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
This PR adds the server routes, not the infrahubctl storage encrypt command or its progress and resume behavior; those are in the separate SDK change. Limit this fragment to the server-side functionality added here.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At changelog/+encryption-existing-files.added.md, line 1:
<comment>This PR adds the server routes, not the `infrahubctl storage encrypt` command or its progress and resume behavior; those are in the separate SDK change. Limit this fragment to the server-side functionality added here.</comment>
<file context>
@@ -0,0 +1 @@
+Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged.
</file context>
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |
| Added the server routes needed to encrypt files stored before storage encryption was enabled, with settings that limit the pace and memory used by the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. |
| botocore.exceptions.ClientError: If the object cannot be written. | ||
|
|
||
| """ | ||
| self._bucket.put_object(Key=name, Body=content, ACL=self.AWS_DEFAULT_ACL, ContentType=self.default_content_type) |
There was a problem hiding this comment.
P2: This unconditional S3 PUT can recreate an object deleted after migration reads it, so the endpoint reports success and resurrects deleted storage. Use an ETag/version-conditional replacement and report a missing or conflict result when it no longer exists.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/storage.py, line 208:
<comment>This unconditional S3 PUT can recreate an object deleted after migration reads it, so the endpoint reports success and resurrects deleted storage. Use an ETag/version-conditional replacement and report a missing or conflict result when it no longer exists.</comment>
<file context>
@@ -99,6 +149,64 @@ def read_head(self, name: str, length: int) -> bytes:
+ botocore.exceptions.ClientError: If the object cannot be written.
+
+ """
+ self._bucket.put_object(Key=name, Body=content, ACL=self.AWS_DEFAULT_ACL, ContentType=self.default_content_type)
+
</file context>
| path = self._storage._path / identifier | ||
| size = path.stat().st_size | ||
| if size > max_size: | ||
| return size, None | ||
| if before_read is not None: | ||
| before_read(size) | ||
| return size, path.read_bytes() |
There was a problem hiding this comment.
P2: The local size check and read use separate path operations, so an atomic replacement can swap in a larger object after stat and before read_bytes, exceeding the memory budget. Open the file first and fstat/read that descriptor.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/storage.py, line 471:
<comment>The local size check and read use separate path operations, so an atomic replacement can swap in a larger object after `stat` and before `read_bytes`, exceeding the memory budget. Open the file first and fstat/read that descriptor.</comment>
<file context>
@@ -235,3 +361,246 @@ def delete(self, identifier: str) -> None:
+ if isinstance(self._storage, InfrahubS3ObjectStorage):
+ return self._storage.get_bounded(identifier, max_size, before_read)
+ if isinstance(self._storage, fastapi_storages.FileSystemStorage):
+ path = self._storage._path / identifier
+ size = path.stat().st_size
+ if size > max_size:
</file context>
| path = self._storage._path / identifier | |
| size = path.stat().st_size | |
| if size > max_size: | |
| return size, None | |
| if before_read is not None: | |
| before_read(size) | |
| return size, path.read_bytes() | |
| path = self._storage._path / identifier | |
| with path.open("rb") as source: | |
| size = os.fstat(source.fileno()).st_size | |
| if size > max_size: | |
| return size, None | |
| if before_read is not None: | |
| before_read(size) | |
| return size, source.read() |
| except NodeNotFoundError: | ||
| return EncryptResult(identifier=identifier, outcome=EncryptOutcome.MISSING) | ||
| except (OSError, botocore.exceptions.ClientError) as exc: | ||
| return _failed_by_storage(identifier=identifier, size=size, reason="the object could not be replaced", exc=exc) |
There was a problem hiding this comment.
P2: The reason "the object could not be replaced" also covers a failure of the confirming header read, after the encrypted copy was already written. Move the inspect_encryption call out of this try block, or give it its own message, so the result does not claim a failed write.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/storage_encryption/migrate.py, line 170:
<comment>The reason "the object could not be replaced" also covers a failure of the confirming header read, after the encrypted copy was already written. Move the `inspect_encryption` call out of this try block, or give it its own message, so the result does not claim a failed write.</comment>
<file context>
@@ -0,0 +1,291 @@
+ except NodeNotFoundError:
+ return EncryptResult(identifier=identifier, outcome=EncryptOutcome.MISSING)
+ except (OSError, botocore.exceptions.ClientError) as exc:
+ return _failed_by_storage(identifier=identifier, size=size, reason="the object could not be replaced", exc=exc)
+ if check.encryption is not ArtifactEncryption.ENCRYPTED:
+ return _failed(
</file context>
| @@ -0,0 +1 @@ | |||
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |||
There was a problem hiding this comment.
P3: The release note says there are settings that limit the pace, but no setting does that. The two settings limit the memory budget and the largest file size, and the client sets the pace. Reword to name the settings that exist.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At changelog/+encryption-existing-files.added.md, line 1:
<comment>The release note says there are settings that limit the pace, but no setting does that. The two settings limit the memory budget and the largest file size, and the client sets the pace. Reword to name the settings that exist.</comment>
<file context>
@@ -0,0 +1 @@
+Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged.
</file context>
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |
| +Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the size of the files and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. |
| @@ -0,0 +1 @@ | |||
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |||
There was a problem hiding this comment.
P3: This release note overstates the checksumless-file behavior: --allow-unverified intentionally encrypts those files without a recorded checksum. Say they are left unchanged by default and document the opt-in exception.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At changelog/+encryption-existing-files.added.md, line 1:
<comment>This release note overstates the checksumless-file behavior: `--allow-unverified` intentionally encrypts those files without a recorded checksum. Say they are left unchanged by default and document the opt-in exception.</comment>
<file context>
@@ -0,0 +1 @@
+Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged.
</file context>
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged. | |
| Added the `infrahubctl storage encrypt` command and the server routes behind it, so an administrator can encrypt the files stored before storage encryption was enabled, with a progress display, stop and resume, and settings that limit the pace and the memory of the API server. Files that no artifact or file object records a checksum for are reported and left unchanged by default; use `--allow-unverified` to encrypt them without checksum verification. |
There was a problem hiding this comment.
1 existing issue remains and 2 new issues found across 14 files (changes from recent commits).
Confidence score: 3/5
- In
encryption_migration.py, an artifact-value write exception can turn the whole batch response into a 500 after the files are already encrypted. Catch and log this failure so the per-object results can still be returned. - In
encryption_migration.py,_encryptcan outlive request cancellation while using the request-scoped database session, so the artifact write may fail when that session closes. Use an independently scoped session or keep the work within the request lifetime. - In
artifact_values.py,_encryptdiscards the artifact counts returned bystore_encrypted_artifact_values, so the admin response won't show that an artifact still needs attention. Preserve those counts in the response.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/infrahub/storage_encryption/artifact_values.py">
<violation number="1" location="backend/infrahub/storage_encryption/artifact_values.py:96">
P2: `store_encrypted_artifact_values` computes the artifact counts, but `_encrypt` discards them. A failed artifact write therefore reaches only the log, and the admin's response does not show that the artifact still reads `plaintext`. Either return these counts to the route and include them in the response, or drop the unused result type.</violation>
</file>
<file name="backend/infrahub/api/storage/encryption_migration.py">
<violation number="1" location="backend/infrahub/api/storage/encryption_migration.py:167">
P2: An unexpected exception in the artifact-value step turns the whole batch response into a 500, even though the files were already encrypted. Catch and log errors from `store_encrypted_artifact_values` so the per-object results are still returned, as the docstring promises. The retry-safe `ALREADY_ENCRYPTED` path would self-heal the stored value on the next pass, so the impact is limited to the response.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Re-trigger cubic
| updated += 1 | ||
| if failed: | ||
| log.warning("The encryption of some artifacts was not stored", branch=branch.name, failed=failed) | ||
| return ArtifactValuesResult(updated=updated, already_correct=len(artifacts) - len(stale), failed=failed) |
There was a problem hiding this comment.
P2: store_encrypted_artifact_values computes the artifact counts, but _encrypt discards them. A failed artifact write therefore reaches only the log, and the admin's response does not show that the artifact still reads plaintext. Either return these counts to the route and include them in the response, or drop the unused result type.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/storage_encryption/artifact_values.py, line 96:
<comment>`store_encrypted_artifact_values` computes the artifact counts, but `_encrypt` discards them. A failed artifact write therefore reaches only the log, and the admin's response does not show that the artifact still reads `plaintext`. Either return these counts to the route and include them in the response, or drop the unused result type.</comment>
<file context>
@@ -0,0 +1,96 @@
+ updated += 1
+ if failed:
+ log.warning("The encryption of some artifacts was not stored", branch=branch.name, failed=failed)
+ return ArtifactValuesResult(updated=updated, already_correct=len(artifacts) - len(stale), failed=failed)
</file context>
| wait_seconds=wait_seconds, | ||
| allow_unverified=allow_unverified, | ||
| ) | ||
| await store_encrypted_artifact_values( |
There was a problem hiding this comment.
P2: An unexpected exception in the artifact-value step turns the whole batch response into a 500, even though the files were already encrypted. Catch and log errors from store_encrypted_artifact_values so the per-object results are still returned, as the docstring promises. The retry-safe ALREADY_ENCRYPTED path would self-heal the stored value on the next pass, so the impact is limited to the response.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/infrahub/api/storage/encryption_migration.py, line 167:
<comment>An unexpected exception in the artifact-value step turns the whole batch response into a 500, even though the files were already encrypted. Catch and log errors from `store_encrypted_artifact_values` so the per-object results are still returned, as the docstring promises. The retry-safe `ALREADY_ENCRYPTED` path would self-heal the stored value on the next pass, so the impact is limited to the response.</comment>
<file context>
@@ -150,7 +164,16 @@ async def _encrypt(
allow_unverified=allow_unverified,
)
- )
+ await store_encrypted_artifact_values(
+ db=db,
+ results=results,
</file context>
With storage encryption on, a customer cannot tell which artifacts are still
stored without encryption, or which encrypted artifacts the current key can no
longer read.
Add a read-only `encryption` attribute to CoreArtifact: plaintext, encrypted
or unreadable, with plaintext as the default. The check treats an artifact
that has no value yet as plaintext.
Add POST /api/artifact/{artifact_id}/encryption. It reads the first bytes of
the stored file, which name the key that sealed it, and reports the value with
a reason (no_key, unknown_key, unsupported_format, truncated or missing). It
does not decrypt the file, so it does not find modified content. When the
result differs from the stored value, it stores the result if the caller may
update artifacts and the branch accepts writes. A missing file leaves the
stored value as it is.
The SDK skips read-only attributes when it saves a node, so the task worker
cannot write the attribute. After it saves an artifact, the worker calls the
endpoint, and the API server is the only writer. A failed call is logged and
leaves the stored value as it was.
Add storage_encryption_enabled to /api/info, which follows the anonymous
access setting, unlike /api/config, which is always public.
The artifact page checks the file when it opens. It shows a warning for a
plaintext artifact while encryption is enabled, a green notice for an
artifact that the check confirmed as encrypted, and an error for an
unreadable one, with the reason when the check gave it. A stored `encrypted`
value alone does not show the green notice, because it can be out of date.
Regenerate the schema, OpenAPI, protocol and frontend type files, document the
check, and add a changelog fragment. The SDK protocol file is generated in the
python_sdk submodule and is not changed here.
Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The code review of the previous commit found two defects. The artifact page showed the green "encrypted" notice for an artifact whose stored value is `encrypted` and whose file is missing from the storage. The check answers with the stored value and the reason `missing` in that case, and the page counted any answer as a confirmation. A missing file now confirms nothing, and an unreadable artifact whose file is missing says so. `read_header` returned None for every S3 client error, so a denied or throttled request was reported as a missing file. Only a not-found error counts as missing now, and other errors are raised. The check endpoint no longer reads the artifact at the `at` time of the request, because it writes at the current time. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The second code review of this change found two gaps in the previous fix. The artifact page warned "stored without encryption" for a plaintext artifact whose file is missing from the storage, because a missing file leaves the stored value as it is. A missing file now gives no notice. An artifact that is already unreadable says that its file is missing, instead of blaming the key. The classification of S3 errors in `read_header` had no test. Add tests with a stand-in S3 storage: not-found errors give None, other client errors are raised, and an empty object gives an empty header. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
Address the review comments of cubic on this change. The artifact page sent the encryption check while a past date was viewed. The check reads the file as it is now and can write the value on the branch, so it does not describe the past. At a past date the page now sends no check and shows no notice. The texts of the unreadable reasons were keyed by any string. They are now keyed by the reasons of the API without `missing`, so a new reason does not compile until it has a text. Add tests for the `unsupported_format` and `truncated` texts. Regenerate `schema/openapi.json` and the frontend API types. They still held an `encrypted` field on the upload response that the code does not have, which failed the `json-schema` check. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The previous commit hid every encryption notice while a past date is viewed, so an artifact stored without encryption showed no warning there. At a past date the artifact page now shows the notice for the value stored at that date, as for a visitor who is not signed in: the warning for a plaintext artifact, the error for an unreadable one without a reason, and no green notice. It still sends no check, and it ignores a check result of the current file. Document this behaviour, and shorten one comment to a single sentence. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
A confirmed encryption needs no action from the user, and the green banner took space on the artifact page only to say that everything is fine. Show a green "encrypted" badge with a lock next to the status badge of the artifact instead. The warning and the error stay banners. The badge appears under the same rule as the banner did: a check of the file confirmed the encryption, and no past date is viewed. Move the rule that picks the notice from a check and the stored value into a hook that the banner and the badge share. Update the documentation and the changelog entry. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The code review found that the notice test "does not reassure about an encrypted artifact while a past date is viewed" passed even when the past-date rule was removed, because the notice never shows the encrypted state. The badge test "shows no badge while a past date is viewed" covers the rule, so remove the notice test. Say in the documentation that at a past date the page shows the warning or the error without the reason, and no badge. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The application query client sets `retry: false` for every query, so the check query does not need to set it again. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
The check needs a signed-in user, so the backend refuses it for an anonymous visitor, and the page sent it anyway. Disable the query when the visitor is not signed in. The page then shows the stored value, as the documentation says. The hook test now asserts the artifact and the branch that the check is sent with, and covers an anonymous visitor. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
…abled
Files stored before storage encryption was enabled stay in plaintext, and
regenerating an artifact does not encrypt them. An administrator can now
encrypt them through three storage routes. The key stays on the API server
and the client sets the pace.
GET /api/storage/objects lists the stored objects in pages. POST
/api/storage/objects/encrypt and POST /api/storage/object/{identifier}/encrypt
encrypt up to 500 objects or one object. For each object the server reads
the header, checks the recorded checksum, seals the file, writes it back
under the same identifier and confirms the header. An object with no
recorded checksum is reported and left unchanged. The routes need the super
admin permission and answer 409 when encryption is not enabled.
A byte budget per API server process
(INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MEMORY_BUDGET) bounds the memory used
while files are encrypted; a request that cannot get memory is answered 429
with Retry-After. A file larger than
INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MAX_SIZE is skipped. The first page of
a listing removes temporary files older than one hour that a killed server
left in local storage.
The OpenAPI schema, the frontend REST types, the configuration reference and
the encryption documentation are updated.
Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
…ncrypts After the migration encrypted their files, the artifacts still stored `plaintext` in `encryption`, because only the task worker and the check route write the value. Readers of the attribute (GraphQL, filters, the generated protocols) were told that encrypted artifacts are plaintext until each artifact page was opened. The migration routes now store `encrypted` on the default-branch artifacts whose current file they encrypted or found already encrypted. One query finds the artifacts of a request and one write is made for each artifact whose value differs. A failed write is logged and does not change the outcome of the file; a new pass over the same files writes the value again. An artifact that exists only on another branch keeps its value until its check runs. The check route and the migration share the helpers that store the value. The OpenAPI schema and the frontend types are regenerated: the description of the listing route changed in the previous commit without them, which fails the json-schema check. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
d7a8c51 to
6349039
Compare
426e49e to
6c90c72
Compare
…ttribute to the CoreArtifact protocols The generated SDK protocols lack the `encryption` attribute of `CoreArtifact`, so `backend-validate-generated` fails and the gate skips the huge-runner jobs. The SDK commit adds it (opsmill/infrahub-sdk-python#1526) and builds on the commit this branch pointed at before. Point at the merge commit once that pull request is merged. Signed-off-by: Simon AURA--SCUSSEL <simon@opsmill.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 10 unresolved issues from previous reviews.
View guided diff | Re-trigger cubic
Merging this PR will not alter performance1 warning
Comparing |
For humans
Customers cannot see which artifacts are stored unencrypted, and cannot encrypt the old files.
Adds an encryption state per artifact with page notices, and routes that encrypt the old files.
Check: the routes, the checksum rules and the memory limit. Stacked on #10746.
Why
Customers who enable storage encryption encrypt only files stored from then on. Without a status they cannot see which artifacts are still stored without encryption. After a key change they cannot see which encrypted artifacts the current key can no longer read. Source: INFP-740.
Files stored before encryption was enabled also stay in plaintext, and regenerating an artifact does not encrypt them. An operator has no way to encrypt them. The routes of this pull request let an administrator do it through the API server, which holds the key, with a client that sets the pace. The client is the
infrahubctl storage encryptcommand: opsmill/infrahub-sdk-python#1535.Base branch:
sas-storage-encryption(PR #10746, not merged). Retarget todevelopafter it merges.Non-goals: a list or filter of unreadable artifacts, key rotation, checking the content of a file.
What changed
encryptiononCoreArtifact:plaintext(default),encryptedorunreadable. The check treats an artifact with no value asplaintext.POST /api/artifact/{artifact_id}/encryption. It reads the first bytes of the stored file. They name the key that sealed it, so the check tells a file sealed with another key, a file with no key available, an unsupported format and a truncated file. It does not decrypt, so it does not find modified content. Reasons:no_key,unknown_key,unsupported_format,truncated,missing./api/infogainsstorage_encryption_enabled. It follows the anonymous access setting, unlike/api/config, which is always public.plaintextand encryption is enabled.encryptedbadge with a lock, next to the status badge: a check confirmed the artifact isencrypted. A storedencryptedvalue alone shows nothing, because it can be out of date.unreadable, with the reason when the check gave it. When the file is missing, anunreadableartifact says so.GET /api/storage/objectslists the stored objects in pages. The cursor follows the last name scanned, so a file that disappears during a scan does not end the listing early.POST /api/storage/objects/encryptencrypts up to 500 objects, andPOST /api/storage/object/{identifier}/encryptone object. The three routes need the super admin permission and answer 409 when encryption is not enabled or the key is missing.fsync, rename; S3: put) and confirms the header. The outcomes areencrypted,already_encrypted,skipped_unverified(no node records a checksum, for example an orphan file),skipped_checksum_mismatch,skipped_too_large,missingandfailed.INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MEMORY_BUDGET(MB, default 600), bounds the memory used while files are encrypted. A request that cannot get memory is answered 429 withRetry-After.INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MAX_SIZE(MB, default 100) is the largest file the migration reads.--dry-runof the client therefore does not leave local storage untouched.encryptedon the default-branch artifacts whose current file is one of them (outcomesencryptedandalready_encrypted): one query for the artifacts of a request and one write for each artifact whose value differs. A failed write is logged and does not change the outcome of the file; a new pass over the same files writes the value again. An artifact that exists only on another branch keeps its value until its check runs.Screenshots
Artifact page, local stack, signed in as an admin. The encrypted state is on commit
2e68819, the others on815e64b. The warning and error banners did not change between the two.How to review
backend/infrahub/storage_encryption/status.pyand, for the routes,backend/infrahub/api/storage/encryption_migration.py,backend/infrahub/storage_encryption/migrate.py,queries.py,artifact_values.pyandmemory_budget.py; thenbackend/infrahub/api/artifact.py(check_artifact_encryption), thenbackend/infrahub/artifacts/encryption.py, thenfrontend/app/src/entities/artifacts/domain/rules/get-artifact-encryption-notice.ts.CoreArtifactis branch-local. Not confirmed: that this shows as a change in a proposed change._postand_request_headers. No other code inbackend/infrahubdoes.POST /api/artifact/generate/encryptionis routed to the check, not to the generation of a definition namedencryption, because the check route is registered first.How to test
test_info_endpointpinned the exact fields of/api/info. Two frontend tests built an/api/inforesponse by hand. A git test fixture now mocks the new worker request.test_artifact_generate_python_newends with an unmatchedGET /api/admin/version. It also happened on the first test of a run before this change. Not root-caused.AccessDeniedas missing fails the S3 test. Without the missing-file branch, 2 frontend tests fail.encryptedon it.a0e74f8, before its commit (warning shown from the stored value, no check request). The badge was checked in Chrome on the code ofb3ca3b8, before its commit.Tests of the routes
backend/tests/unit/storageandbackend/tests/unit/storage_encryption: 373 passed. Component teststest_60totest_65, single process: 49 passed. The whole backend unit suite with-n 4: 5,581 passed.test_migrate_parity.py,test_64).test_66). With the artifact step switched off, 5 of them fail withplaintextwhereencryptedis expected. On a local stack with 1,000 artifacts (3,150 files), the SDK command left all 1,000 valuesplaintextbefore this commit andencryptedafter it, with every file encrypted and 0 failed downloads. The migration took 4.5 s before and 8.5 s after (3 runs each, about 4 ms and 3 database statements per updated artifact). Not measured: S3 and 50,000 artifacts.Impact & rollout
/api/info, two new settings.INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MEMORY_BUDGET(600) andINFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MAX_SIZE(100). With several API server processes the worst-case memory is the budget times the number of processes.CoreArtifactlives in thepython_sdksubmodule. The change is in opsmill/infrahub-sdk-python#1526, which is not merged. The submodule pointer is bumped to its commit (4fbf74e1, which builds on the previous pointer), and must be bumped again to the merge commit once Added Playwright Testing #1526 is merged.Not done or not verified
encryptionvalue after the migration, until its check runs (the artifact page, or the worker when the artifact is generated). Supporting it needs a branch lookup, per-branch permission and status checks, and more writes.426e49e68c) and the regenerated schema were not reviewed by thecode-revieworsecurity-reviewskills. The reviews below cover60ac20562f(code review) and45b41d53e6(security review).The branch was rebased on the rebased #10746 and on
develop(ee2d2dc402) on 2026-11-17. The commit hashes in the review sections below are those before the rebase. The content did not change: the only conflict was one import line inbackend/infrahub/storage.py.Code review
The
code-reviewskill ran at levelhigh, in eight rounds, each by an agent other than the author. In round 2 the skill reviewed the whole branch, so the agent kept only the findings that touch the fix or that it could verify. Round 4 reviewed a version that was not yet committed. Rounds 6 and 7 added mutation tests on a copy of the code.b86987248dd723815e64b4781b77, before its commit4781b77815e64b..b3ca3b82e6881971d011bfde5c63removes one query option that the query client already sets. It had no review of its own.Important findings:
48dd723.815e64b.815e64b.useQueryand proved nothing. Both fixed in4781b77.a0e74f8, which shows the stored value at a past date.a0e74f8.a0e74f8after a decision: it shows the stored warning or error.2e68819. The badge test covers the rule.Minor, fixed: every S3 client error counted as a missing file, and the endpoint read the artifact at the
attime of the request while it writes at the current time.Minor, left as they are: the
atparameter is accepted and ignored; an S3 store that hides missing keys from callers without list permission gives an error instead ofmissing; the worker uses two private SDK methods; the worker only logs a failed call; the page sends a check even when encryption is off; two functions repeat the same permission logic; the encryption detection rule is in two places; the notice component and the rule both handle a missing file; the S3 tests use hand-written fakes; no component test for anunreadableartifact whose file is missing; the endpoint returns an error when S3 fails for another reason.Minor, left as they are (rounds 4 to 7): the query key has no date, so after a past date is cleared the earlier check result shows until the new check answers; the badge has the same green as the
readybadge and only a tooltip explains it; the past-date condition is in the query hook and in the notice hook; the documentation does not say that a storedencryptedvalue shows nothing at a past date, and the warning at a past date follows the setting now; no notice-level test for a storedencryptedvalue at a past date; the removal of theencryptedfield ofUploadResponsefrom the schema is outside the UI change, and the backend model never had it. Round 8: a check result stays displayed in a tab after the user signs out in another tab, until a reload; no test for an anonymous visitor who signs in; the hook test replaces the whole authentication module.Code review of the routes
The
code-reviewskill ran once at levelhigh, by an agent other than the author, on commit60ac20562f, which is the first version of the routes. The fixes were amended into45b41d53e6and were not reviewed again. The skill reviewed the right checkout.0 critical, 2 important, 9 minor.
Important, both fixed:
stat()made the listing return a short page with no cursor, so the client stopped and every later file stayed plaintext while the run looked finished. The cursor now follows the last name scanned (local and S3).Minor, fixed: the
--dry-runtext, raw exception text inreason, a note that the two checksum queries must stay in step.Minor, left as they are:
putcan recreate an object deleted between the read and the write./or..makes the whole batch answer 422 (shared bucket with prefixes only).Security review
The
security-reviewskill ran from a clone whoseorigin/HEADwas set tosas-storage-encryption, so its diff is this change. It ran in seven rounds, each by an agent other than the author. In every round the agent did not start the skill's sub-tasks and read the diff by hand. In rounds 3 to 7 the skill reviewed the worktree and not the clone, so the agent discarded its input and applied its method by hand to the diff of the commits.b869872,48dd723815e64b4781b77a0e74f8815e64b..b3ca3b82e6881971d011bLow, left as they are:
encryptioncannot be generated through the API (listed above).storage_idis writable, so a user who may update artifacts can learn from the check whether a server path exists. The content endpoint already reads that path.encryptedorplaintext. This is less visibility, not less protection.Dropped after checks: forged requests from another site, setting the value from a client, injection in the worker URL, a mismatch of the stored value by someone who can write to the storage, and XSS in the notice text.
Security review of the routes
The
security-reviewskill ran once, by an agent other than the author, on45b41d53e6. Its diff was this commit; the agent applied its method by hand.0 high, 0 medium, 1 low (7 findings dropped as unverified or speculative).
Low, left as it is: the checksum check proves that the content equals a recorded checksum, and the lookup accepts a checksum from any node on any branch. Someone who can already write to the storage and can create an artifact node could record the checksum of a changed file, and a later migration by a super admin would seal the changed content. The cost is that the checksum does not protect against tampering by someone with both rights. The migration never lets a file swapped during its own read-and-write window be sealed, and never rewrites a file that is already encrypted. A possible mitigation is to take checksums only from the default branch.
Checklist
uv run towncrier create ...)