Repository navigation
feat: back up and restore external databases on Kubernetes - #172
Conversation
Deploying infrahub-ops-cli with
|
| Latest commit: |
9a61410
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4dab6500.infrahub-ops-cli.pages.dev |
| Branch Preview URL: | https://007-external-database-backup.infrahub-ops-cli.pages.dev |
Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This supplies one — a short-lived Job in the namespace acting as the vendor's own backup client — so the existing execution, streaming and copy primitives work unchanged and a deployment whose databases are all internal behaves exactly as before (FR-015). Capture runs `neo4j-admin database backup --from=…` or `pg_dump` in that workload and streams the artifact through the existing pipeline, so an artifact taken externally is byte-compatible with an internal one and restores by the same code. Where a database lives is settled at a gate from positive evidence: a deployment query that fails is an error naming the permission, never a silent switch to the external path, so a permission problem cannot create workloads or pull images (FR-002). Everything that would yield an unusable artifact is caught before data moves — a utility older than the server, an external Community database, missing workload permissions, a restore artifact the server cannot read. Restore inverts the data path, because the server fetches the artifact itself. The raw backup is staged in the configured object store, proven readable by an independent client, seeded over Bolt, and then confirmed online by polling: an accepted seed is not a completed one (FR-021). It requires its own authorisation, distinct from --force and from enabling scheduled restore (FR-009). Nothing outlives the run. The workload carries its own deadline, its credential Secret is garbage-collected with it, and a reaper removes strays left by earlier runs (FR-011, FR-023). Connections are encrypted and verified by default, with one operator opt-out (FR-029). Incomplete captures are removed rather than retained, completeness is read from the backup command's output rather than its exit status, and metadata additions are additive so older releases keep restoring new artifacts (FR-012, FR-016, FR-024). Kubernetes resource ownership is decided in one place from label evidence and declared release prefixes, replacing the substring matches that could scale an unrelated workload to zero. Spec, research, contracts and task ledger are archived under specs/archive/007-external-database-backup/. The decisions are recorded as ADRs 0008-0016, with operator documentation under docs/docs/backup/ and developer notes under dev/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bd4d482 to
3f27cdb
Compare
There was a problem hiding this comment.
2 issues found across 99 files
Not reviewed (too large): specs/archive/007-external-database-backup/tasks.md (~441 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md">
<violation number="1" location="specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md:95">
P3: E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.</violation>
</file>
<file name="src/internal/app/backup_containers.go">
<violation number="1" location="src/internal/app/backup_containers.go:293">
P2: The quiesce and restart waits share one deadline across all services. If the first service consumes the window (a stuck pod terminating, or a slow image pull), every later service is checked with the deadline already passed and is reported as not down / not running without ever being polled. Compute the deadline per service (or subtract elapsed time before each service's loop) so each stopped/restarted workload gets the full window.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
|
||
| ### 4a. Architecture Soundness | ||
|
|
||
| **E6 🎯** — Pod resolution falls back to **substring matching on pod names** |
There was a problem hiding this comment.
P3: E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md, line 95:
<comment>E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.</comment>
<file context>
@@ -0,0 +1,297 @@
+
+### 4a. Architecture Soundness
+
+**E6 🎯** — Pod resolution falls back to **substring matching on pod names**
+(`environment_kubernetes.go:305-313`, and the same fallback in `getAllPodsWith`). The transient pod
+must contain the canonical service name for that fallback to find it — but the same property makes it
</file context>
There was a problem hiding this comment.
Done in 3584a53. Kept the point-in-time finding and added a dated resolution note naming the current locations (podsForServiceWith, claimedPodNamesWith, parseLabelledPods applying withoutTransientPods for FR-028).
3f27cdb to
8df6a24
Compare
…ating a database deploymentClaimsService reads Deployments and StatefulSets only once no pod declares the service, which is exactly when they decide the answer: a StatefulSet scaled to zero is internal only because its workload says so. A failed workload listing was logged at debug level and skipped, so a forbidden or unreachable listing returned "external" for a question that was never answered. FR-002 names permission denied as a failure to query, not as evidence. listAllWorkloadsWith now returns the error, deploymentClaimsService and locateServiceWith carry it into undeterminedLocation, and the RBAC hint names deployments and statefulsets beside pods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
minio/minio is no longer published on Docker Hub or quay.io, so every S3 e2e test errored in fixture setup with ImageNotFound. pgsty/minio is a community build of the same server; the fixture command and health endpoint are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slow external stream that kept making progress never tripped the idle timeout, so the transient pod's activeDeadlineSeconds could end it. The pod's removal ends kubectl's stdout like a clean finish, and kloset discards the reader's Close error, so the truncated archive was committed as a complete snapshot. execReadCloser now waits for the producing command on EOF and returns its failure from Read, which kloset records as a source error and the snapshot commit refuses. The external Neo4j and PostgreSQL streams are also held to the capture's bound as a whole, the full-bound slot the capture workload's deadline already budgets for the artifact leaving the pod, so the tool ends a slow transfer before the cluster does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RestoreBackup settled where each database lives from --exclude-taskmanager alone, before the archive was read. An archive without the task-manager database was then refused, or given a transient workload, for an external PostgreSQL the restore never writes. The gate now runs after the metadata is parsed, on taskManagerIncluded && !excludeTaskManager, and still before the edition probe, the wipe and the stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s in every restore path e200950 moved RestoreBackup's gate after the archive metadata is read. The Plakar restore and the S3 leg of restore --latest still gated the task-manager database on --exclude-taskmanager alone, before the backup was read, so a backup without that database was refused, or given a transient workload, for an external PostgreSQL the restore never writes. The Plakar group restore now gates once the group is chosen, on whether it holds a postgres snapshot; the single-snapshot restore gates once the snapshot's component is read, before the edition probe. The S3 leg gates only Neo4j before the download, which every archive carries, and leaves the task manager to RestoreBackup's gate after the metadata is read; the second pass over Neo4j prepares nothing again. All gates still run before anything is stopped or wiped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… way libpq does statedKeywordFields split the keyword/value form on whitespace, so a quoted value holding a space, such as sslrootcert='/etc/ssl/my certs/ca.crt', was cut at the space, and whitespace around "=" lost the setting. Discovery then recorded a truncated host or certificate path for the external task-manager database. The form is now parsed as libpq parses it: whitespace around "=", single- quoted values, and backslash escapes for \' and \\. A string libpq would refuse (an unterminated quote, a keyword without "=") is an error, and discovery reads nothing from it rather than part of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cannot be located environment detect resolved both databases through resolveDatabaseTargets, which stops at the first location query that fails. One failed query hid the other database from the report. The report now resolves each database on its own through databaseTargetWith and logs a per-database warning; the run gate still stops at the first unknown location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… query The fake backend answered every cypher-shell call with success, so the subtest exercised the parse of a successful answer and never the assume-not-clustered branch it is named for. The fake can now fail cypher-shell calls, and the subtest asserts the query was issued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Run golangci-lint v2.13.2 in CI, matching the dev-setup pin. - Share the build tag between GO_TAGS and the lint target. - Clear INFRAHUB_EXTERNAL_DB_INSECURE_TLS in the CLI test environment. - Correct the spec directory path in plan.md, reconcile the open-task count in the implementation report, note where critique E6 was resolved, and align the backup artefact wording with FR-005. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c560786 made execReadCloser wait for the producing command when the stream reaches EOF. Waiting closes the command's pipe, and kloset reads once more after an EOF, so that read returned "file already closed". kloset recorded it as a source error and every Plakar Neo4j snapshot was refused, on Docker and Kubernetes alike. Once the command has been waited for, Read now returns EOF. The new test streams a real process through kloset, which the fake pipes in the existing tests could not reproduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…paths - restore --latest --s3 resolves database credentials before gating, so the external Neo4j target carries the default database and user - --reset-deployment-id runs through the restore pod when Neo4j is external - a single Plakar postgres-snapshot restore restarts every service it stopped, and gates only the database it writes - the restore pod's deadline covers every bound the pod is held through - docker IsRunning treats a service the compose file lacks as not running - the Prefect chart's component=server label no longer claims a pod for infrahub-server - resolveJobPod falls back to polling when kubectl predates --for=create - drop statedPrefectSSLRootCert, which only tests called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 12 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…deadline When Neo4j is external the deployment-ID reset runs through the restore workload, under the probe bound and retried up to ten times. That worst case (about 50 minutes at the defaults) was not in the pod's activeDeadlineSeconds, and the 10-minute margin cannot absorb it, so the cluster could remove the pod during the reset and fail a restore that had already completed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The polling fallback for kubectl without --for=create ran each listing under the control bound, independently of the poll's deadline. A listing started just before the deadline could return after it, stretching pod resolution past its ten-minute bound and accepting a pod found late. Each listing now runs through queryWithin with the time left, and none is issued once the deadline passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/api/config answers before infrahub-server can reach its database again after the database pod reset between tests, so the next test's first SDK call (the schema fetch) could get a 503. Wait for /api/schema as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docker The Community streamed load resumes a Neo4j that then finishes its shutdown and is restarted by its restart policy. `docker compose start infrahub-server` checks the `service_healthy` dependency once and refuses an exited database, so the post-restore start raced that restart and failed with a bare "exit status 1". All restore paths now start infrahub-server and task-worker through one helper that, on Docker, waits (bounded, 5 min) for every database container to be running and freshly healthy, starting it if it stays exited. DockerBackend.Start and Stop include compose's output in their errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
The five-minute wait only checked its deadline between readings, so a hung docker compose ps, docker inspect or docker compose start held it indefinitely. Each call now runs with the time left before the deadline and a call cut short returns a timeout error naming it. runCommandContext also bounds the wait for a killed command's output pipe, which a child such as the compose plugin keeps open after the docker CLI is killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its URL /api/schema can answer while infrahub-server still cannot reach Neo4j after the database pod reset between tests, so the next test's first mutation failed with 'Unable to connect to the database'. Wait for a GraphQL query that reads the database instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/internal/app/docker_service_ready_test.go">
<violation number="1" location="src/internal/app/docker_service_ready_test.go:170">
P3: These fake Docker scripts leave each `sleep 30` child running for 30 seconds: `CommandContext` kills only the shell, and `WaitDelay` abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| bin := t.TempDir() | ||
| // Not `exec sleep`: the sleep is a child that keeps the output pipe open | ||
| // after the killed shell, as the compose plugin does under the docker CLI. | ||
| script := "#!/bin/sh\nsleep 30\n" |
There was a problem hiding this comment.
P3: These fake Docker scripts leave each sleep 30 child running for 30 seconds: CommandContext kills only the shell, and WaitDelay abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/internal/app/docker_service_ready_test.go, line 170:
<comment>These fake Docker scripts leave each `sleep 30` child running for 30 seconds: `CommandContext` kills only the shell, and `WaitDelay` abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.</comment>
<file context>
@@ -160,3 +160,44 @@ func TestDockerBackendStartIncludesComposeOutput(t *testing.T) {
+ bin := t.TempDir()
+ // Not `exec sleep`: the sleep is a child that keeps the output pipe open
+ // after the killed shell, as the compose plugin does under the docker CLI.
+ script := "#!/bin/sh\nsleep 30\n"
+ if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(script), 0o755); err != nil {
+ t.Fatalf("writing fake docker = %v", err)
</file context>
There was a problem hiding this comment.
Valid point — with the process-group kill, the fake scripts' sleep 30 children now die with the group when the timeout fires, so no orphans remain after the tests, in d4bb6b5.
…losed The child's context was cancelled only after cmd.Wait returned, so a stream that hit its idle timeout or deadline closed stdout and then waited on a child blocked in database I/O, which never notices the closed pipe. Closing the stream now cancels the child first. Children run in their own process group, so cancelling also stops what an `sh -c` pipeline started, and WaitDelay bounds the wait for their output pipes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
copyAcrossTransientWorkload returned before taking externalDBBound when the target was a local workload, so a copy into or out of the scratch directory on a slow or stalled filesystem could hold a backup or restore past the bound every other external operation runs under. Local copies now take a context and stop between reads once it is done, and the seam runs them under the same bound, returning the timeout at the bound even when a copy is stuck inside one read or write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The external suites are Python, so a change to them alone skipped the job that runs them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A missing Secret key is fixed by restoring the key; restarting the pods does not help. - Direct mode also needs a temporary directory with a plain path. - FR-031's refusal of a development build applies only to runs that create a transient workload. - ADR 0008: the external path adds write permissions to those the Kubernetes path already needs. ADR 0018: the cleanup guarantee depends on the Job, TTL and garbage-collection controllers. - The changelog says only named Secret keys avoid the secrets permission. - The knowledge doc: a failed pod-spec read is not remembered, and the image to mirror is the one --external-db-image selects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
1 existing issue remains and no new issues found across 10 files (changes from recent commits).
You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
With --k8s-namespace set and the namespace check refused, detection fell through to Docker. In CI this picked up another job's Compose project on the runner, and the run started backing it up; a restore would have written to it. An explicit target now pins its backend, and that backend's detection failure is the run's failure (tasks.md T156). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
- e2e-tests-k8s needs go-lint directly, so a lint failure stops it. - ADR 0008 names the scale resources as the only write kubectl scale makes. - The configuration reference describes what happens when both --project and --k8s-namespace are set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the conflicts between the external database work and the fixes released in v2.4.0 and v2.4.1: - restore: add --neo4j-cluster-online-timeout to newRestoreCommand - k8s: try infrahub/service first in serviceLabelKeys, then the Neo4j Helm chart label for `database`, so the Neo4j service does not resolve to CloudNativePG pods (f673202, 1e0e522) - backup and restore: record workload replica counts before anything is stopped (9bbb849), on the PR's edition gate - Neo4j edition: keep requireDeterminedEdition and add the probe error to it; remove main's detectBackupNeo4jEdition, which did the same - restore: follow main's rule that an unknown edition stops the restore of an archive without a recorded edition (84dc735), and update the PR test that expected a fallback - remove the duplicate runCommandSeparate and newTestKubernetesBackend - e2e: use main's Garage S3 server and authenticated /api/schema wait, then the PR's database query wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alpine removed openjdk21-jdk and openjdk21-jmods 21.0.12_p8-r0 when it published 21.0.13.0.3_p1-r0, so the image build stopped at the java-runtime stage. Both e2e-tests-k8s jobs on the merge of main failed there. scripts/check-image-tools.sh passes on the rebuilt image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Kubernetes case ran against one Neo4j server, so the cluster restore flow and the external path against a cluster had no end-to-end coverage. Two topologies now run a three-member Neo4j cluster, built the way Infrahub's HA example builds it: three Neo4j Helm releases and a headless service, with Infrahub connecting through the neo4j protocol. - internal: the cluster runs in the Infrahub namespace with the infrahub/service=database label. A tarball round trip checks that the restore takes the cluster flow and that the database is online on every member afterwards. - external: the cluster runs in its own namespace, so the tool takes the external path. One case checks what the capture records for the cluster address, and one runs the full round trip, where the restore seeds every member from the object store. The new e2e-tests-k8s-neo4j-cluster job runs each topology on its own runner with the enterprise chart. Both topologies passed locally on a docker-driver vcluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mypy cannot index Popen.args, which python-lint reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Address the review of the Neo4j cluster tests: - Create the vcluster with three nodes when a session declares a cluster topology, and keep the chart's anti-affinity, so each member runs on a node of its own. The install fails when the members share a node. - Stage external restores in the session's Garage container (s3_server), attached to the vcluster's Docker network, instead of a MinIO container started by CI. - Build the external topology in the fixtures: the address, the credentials, the flags and the infrahub-backup image. The CI job only declares the topology and runs pytest. Add quality checks: - After a restore, read every member on its own and check that all hold the same graph. - Fail a session that declares a topology and passes none of its cases. - Dump the cluster namespaces' logs when a cluster case fails. - Upload the JUnit results of the cluster jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…space Add a case to the external cluster suite that runs infrahub-backup in a pod in the Infrahub namespace, the way the Helm chart runs it. The run gets no --external-db-image, uses the tools of its own image (direct mode, ADR 0017) and creates no transient Job. Its service account holds only the direct-mode permissions. The case checks the backup and the restore logs, that every member is online with the same graph, that the pre-backup data is back, and that no transient object is left. Move the direct-mode log lines into conftest so both suites share them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how to back up and restore an external database with the infrahub-backup Helm chart. Add a section to the external database guide that lists the chart's externalDatabase values and the environment variable each one sets, the settings the chart has no value for (scratch size, a larger temporary volume), the restore requirements on the S3 bucket, and one values example for a backup and one for a restore. Link it from the prerequisites of the Kubernetes backup and restore guides, and point the cluster online timeout troubleshooting at the new chart value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/docs/backup/kubernetes-restore.mdx">
<violation number="1" location="docs/docs/backup/kubernetes-restore.mdx:29">
P2: The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| - Access to modify Helm values for your Infrahub deployment | ||
| - S3 credentials with read access to the backup bucket | ||
| - Network egress from the restore pod to the Kubernetes API server (see [Network access to the API server](./kubernetes-backup.mdx#network-access-to-the-api-server)) | ||
| - If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket |
There was a problem hiding this comment.
P2: The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/backup/kubernetes-restore.mdx, line 29:
<comment>The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.</comment>
<file context>
@@ -26,6 +26,7 @@ Before restoring:
- Access to modify Helm values for your Infrahub deployment
- S3 credentials with read access to the backup bucket
- Network egress from the restore pod to the Kubernetes API server (see [Network access to the API server](./kubernetes-backup.mdx#network-access-to-the-api-server))
+- If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket
## Step 1: Identify the backup to restore
</file context>
| - If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket | |
| - If Neo4j or PostgreSQL is not managed by this Infrahub deployment, even when it runs in the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…23 ships Alpine removed postgresql18-client 18.6-r0 when it published 18.7-r0, so the image build stopped at the apk add step. e2e-tests-k8s (enterprise) on the merge of main failed there. scripts/check-image-tools.sh passes on the rebuilt image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| ```bash | ||
| infrahub-backup create \ | ||
| --k8s-namespace infrahub \ | ||
| --external-db-image registry.example.com/opsmill/infrahub-backup:2.3.0 \ |
There was a problem hiding this comment.
I think it doesn't make sense to have that flag, because users in this case will already run infrahub-backup through their registry using the helm chart
We really need to have integration and testing in infrahub-helm before merging this PR
|
|
||
| ## Consequences | ||
|
|
||
| - Customer prerequisites that no earlier feature had: each Neo4j server needs a route to the |
…eased image [IFC-3553] infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub namespace, and the chart now uses it. Remove the build of an image from opsmill/infrahub-backup#172: the Jobs run the image of the chart's appVersion. Move the test from the manual marker to the backup marker, so the e2e-infrahub-backup CI job runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce (#125) * feat(infrahub-backup): support databases outside the Infrahub namespace [IFC-3553] Add externalDatabase values for a Neo4j or PostgreSQL database that runs outside the Infrahub namespace. Each one sets the matching infrahub-backup environment variable: allowRestore (restore containers only), insecureTLS, timeout, scratchSize, neo4jAddress, neo4jBackupPort and postgresAddress. scratchVolume mounts a volume at /scratch and sets TMPDIR to it. A restore by restore.storage.s3.key now also sets INFRAHUB_S3_BUCKET, INFRAHUB_S3_PREFIX and INFRAHUB_S3_ENDPOINT. A restore of an external Neo4j database copies the archive there for the Neo4j servers to download. Add restore.options.neo4jClusterOnlineTimeout, which passes --neo4j-cluster-online-timeout (infrahub-backup 2.4.1 or later). All new values are empty by default, so the rendered chart does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(infrahub-backup): back up and restore an external Neo4j through the chart [IFC-3553] Add a manual e2e test that installs a Neo4j 2025.10.1 Enterprise server in its own namespace and Infrahub with neo4j.enabled=false pointed at it. The backup and restore Jobs of the chart must run the database tools in their own container, create no Job and no Secret, write their temporary files to an ephemeral PersistentVolumeClaim, and bring back the data from before the backup. The test needs an infrahub-backup image built from opsmill/infrahub-backup#172, so it is marked manual until that change is released. Move the Garage setup into deploy_garage so both backup tests use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(infrahub-backup): use infrahub-backup 2.4.1 and document restore bucket access [IFC-3553] Bump appVersion to 2.4.1. restore.options.neo4jClusterOnlineTimeout passes --neo4j-cluster-online-timeout, which infrahub-backup 2.4.0 does not have, so a restore with this value failed with the default image. Document that a restore of an external Neo4j database writes the staged artifact to restore.storage.s3.bucket, also with restore.storage.s3.key. The credentials must be able to write to that bucket, and the Neo4j servers must be able to read from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(infrahub-backup): remove externalDatabase.scratchVolume [IFC-3553] The chart does not need its own value for the volume of the temporary files. An operator who needs a larger volume can add it with extraVolumes and extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now says so. The manual e2e test no longer mounts a scratch volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(infrahub-backup): run the external Neo4j test without --external-db-image [IFC-3553] The backup and restore Jobs run in the Infrahub namespace, so infrahub-backup runs the database tools in its own container and creates no Job. It checks the image of that Job only when it creates one, so the flag is not needed. The test now runs the chart as an operator does, with no extra arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(infrahub-backup): set up the external database namespace in a fixture [IFC-3553] Move the namespace and the Garage of the external Neo4j test into the external_db_s3_k8s fixture, as s3_k8s does for the community deployment. infrahub_external_neo4j_k8s now depends on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(infrahub-backup): remove externalDatabase.scratchSize [IFC-3553] The chart no longer has a value for the free disk space of a capture of an external database. An operator who needs it sets INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025 server needs it, because the server does not report its store size. The manual e2e test now sets the variable with extraEnv, in a values file that also keeps the S3 CA settings: a --set on extraEnv would replace the list of the CA values file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(infrahub-backup): use infrahub-backup 2.6.0 [IFC-3553] Set appVersion to 2.6.0, the first infrahub-backup release that backs up and restores a database outside the Infrahub namespace, which the externalDatabase values need. Replace the 2.4.1 changelog fragment, which infrahub-backup-1.6.0 already released, with one for 2.6.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(infrahub-backup): run the external Neo4j test in CI with the released image [IFC-3553] infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub namespace, and the chart now uses it. Remove the build of an image from opsmill/infrahub-backup#172: the Jobs run the image of the chart's appVersion. Move the test from the manual marker to the backup marker, so the e2e-infrahub-backup CI job runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(infrahub-observability): record every trace in the observability e2e test Infrahub 1.12 adds trace.sample_ratio, which records 10% of new traces by default. The trace of the probe request of test_grafana_can_query_infrahub_metrics_and_traces was then usually not recorded, so Tempo had no trace to find and the test timed out. Set INFRAHUB_TRACE_SAMPLE_RATIO=1 on the Infrahub server in the fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Daniel Pardo <daniel@opsmill.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This branch gives those deployments the same
infrahub-backup create/restoreround trip everyone else has, with no manual steps on the database hosts beyond documented one-time prerequisites, and leaves fully-internal deployments byte-for-byte unchanged.Key Changes
neo4j-admin database backup --from=…orpg_dumpthere, and streams the artifact through the existing pipeline. Artifacts are byte-compatible with internal ones and restore by the same code.--allow-external-restore/INFRAHUB_ALLOW_EXTERNAL_RESTORE), distinct from--forceand from scheduled restore.redis-cacheto zero.make lintwas missing the build tag the Makefile defines and passes to both test and vet,make dev-setuppinned a v1 that cannot read this repo's v2 config, and a linter built by an older Go than the one in use fails every package on typecheck. All three are fixed and documented.Related Context
specs/archive/007-external-database-backup/(archived with its extraction record). User Story 1 (P1 round trip) and User Story 2 (P2 diagnosable refusals) are delivered; User Story 3 (operator-host execution) stays deferred as specified.dev/adr/.neo4j.enabled=false, Neo4j 2025.10.1 Enterprise, MinIO as the artifact store) passed end to end; its residual findings are ledgered as T156–T161 in the archivedtasks.md.Documentation Updates
docs/docs/backup/external-databases.mdx(prerequisites, flags, authorisation, troubleshooting),docs/docs/backup/external-database-direct-to-cloud.mdx, and the flags indocs/docs/reference/configuration.mdx.dev/knowledge/external-database-backup.md,dev/knowledge/container-image-user-semantics.md,dev/guidelines/kubernetes-backend.md,dev/guidelines/implementation-verification.md, ADRs 0008–0016, andAGENTS.mdsections for the new files.Test Plan
make test(orgo test -tags untested_go_version ./src/...): unit coverage for endpoint resolution, both version schemes, metadata compatibility, completeness derivation, ownership decisions, transient-workload manifests and lifecycle. All green.make linton Linux, orgolangci-lint run --build-tags untested_go_version ./src/...on macOS, wheretools/neo4jwatchdogcannot build. Reports 0 issues. Re-runmake dev-setupfirst if the Go toolchain has moved since the linter was installed.tests/e2e/test_k8s_external_neo4j.pyandtests/e2e/test_k8s_external_both.pyagainst a cluster with the database(s) outside the Infrahub namespace;tests/e2e/test_docker_external_compat.pyfor the unchanged Docker path.infrahub-backup createandrestorebehave and lay out artifacts exactly as before (FR-015).🤖 Generated with Claude Code
Assisted-by: speckit-opsmill-auto unversioned
Assisted-by: speckit-opsmill-implement unversioned
Assisted-by: speckit-opsmill-extract unversioned
Assisted-by: opsmill-dev-commit 0.1.0
Assisted-by: opsmill-dev-pr 0.2.0
Summary by cubic
Adds backup and restore for Kubernetes deployments whose Neo4j Enterprise or PostgreSQL database runs outside the cluster.
infrahub-backup createandrestorepreviously could not reach those databases; they now use the vendor's backup client in a transient in-namespace Job and stream artifacts through the existing pipeline. Fully internal deployments are unchanged.New Features
infrahub-backupimage, which shipsneo4j-adminand the PostgreSQL client, so no vendor image pull is needed; when the tool itself runs as a pod in the namespace, direct mode runs the tools in-process and creates no Job or Secret.secretKeyRefreferences exist, falling back to a run-owned Secret otherwise.Bug Fixes
envFrom-backed variables, and the PostgreSQL keyword/value connection form is parsed like libpq.--k8s-namespaceor--projectset, a refused or missing namespace fails the run with kubectl's message instead of falling through to a Docker Compose project.--external-db-timeout.Written for commit 304756b. Summary will update on new commits.