Skip to content

feat: back up and restore external databases on Kubernetes - #172

Merged
danipl merged 64 commits into
mainfrom
007-external-database-backup
Nov 16, 2026
Merged

danipl merged 64 commits into
mainfrom
007-external-database-backup

Conversation

@BeArchiTek

@BeArchiTek BeArchiTek commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This branch gives those deployments the same infrahub-backup create / restore round trip everyone else has, with no manual steps on the database hosts beyond documented one-time prerequisites, and leaves fully-internal deployments byte-for-byte unchanged.

Key Changes

  • Operators with external databases can back up and restore with the commands they already use. The tool creates a short-lived pod in the namespace as the vendor's own "backup client", runs neo4j-admin database backup --from=… or pg_dump there, and streams the artifact through the existing pipeline. Artifacts are byte-compatible with internal ones and restore by the same code.
  • Where a database lives is decided from positive evidence. A deployment query that fails is an error naming the permission, never a silent switch to the external path, so a permission problem cannot create workloads or pull images.
  • Every condition that would yield an unusable artifact is caught before data moves: a utility older than the server, an external Neo4j Community database, missing workload permissions, a restore artifact the server cannot read.
  • Remote Neo4j restore seeds from an object-store URI and is confirmed, not assumed. The tool polls the database until it is online and fails the run otherwise; "accepted" is not "restored".
  • Destructive restore into an unmanaged database needs its own authorisation (--allow-external-restore / INFRAHUB_ALLOW_EXTERNAL_RESTORE), distinct from --force and from scheduled restore.
  • Nothing outlives the run. The transient pod carries its own deadline, its credential Secret is garbage-collected with it, and a stray reaper removes leftovers from earlier runs. Connections are encrypted and verified by default with a single operator opt-out.
  • Incomplete captures are never retained, completeness is read from the backup command's output rather than its exit status, and metadata additions are additive, so older releases keep restoring new artifacts.
  • Kubernetes resource ownership is decided in one place from label evidence and declared release prefixes, replacing the substring matches that could scale another product's redis-cache to zero.
  • The linter runs again. make lint was missing the build tag the Makefile defines and passes to both test and vet, make dev-setup pinned a v1 that cannot read this repo's v2 config, and a linter built by an older Go than the one in use fails every package on typecheck. All three are fixed and documented.

Related Context

  • Spec, research, contracts and task ledger: specs/archive/007-external-database-backup/ (archived with its extraction record). User Story 1 (P1 round trip) and User Story 2 (P2 diagnosable refusals) are delivered; User Story 3 (operator-host execution) stays deferred as specified.
  • Decisions recorded as ADRs 0008–0016 under dev/adr/.
  • First real Kubernetes round trip (minikube, chart 4.33.2 with neo4j.enabled=false, Neo4j 2025.10.1 Enterprise, MinIO as the artifact store) passed end to end; its residual findings are ledgered as T156–T161 in the archived tasks.md.

Documentation Updates

  • Operator docs: docs/docs/backup/external-databases.mdx (prerequisites, flags, authorisation, troubleshooting), docs/docs/backup/external-database-direct-to-cloud.mdx, and the flags in docs/docs/reference/configuration.mdx.
  • Developer docs: dev/knowledge/external-database-backup.md, dev/knowledge/container-image-user-semantics.md, dev/guidelines/kubernetes-backend.md, dev/guidelines/implementation-verification.md, ADRs 0008–0016, and AGENTS.md sections for the new files.

Test Plan

  • make test (or go test -tags untested_go_version ./src/...): unit coverage for endpoint resolution, both version schemes, metadata compatibility, completeness derivation, ownership decisions, transient-workload manifests and lifecycle. All green.
  • make lint on Linux, or golangci-lint run --build-tags untested_go_version ./src/... on macOS, where tools/neo4jwatchdog cannot build. Reports 0 issues. Re-run make dev-setup first if the Go toolchain has moved since the linter was installed.
  • End-to-end: tests/e2e/test_k8s_external_neo4j.py and tests/e2e/test_k8s_external_both.py against a cluster with the database(s) outside the Infrahub namespace; tests/e2e/test_docker_external_compat.py for the unchanged Docker path.
  • Manual check on a fully-internal deployment: infrahub-backup create and restore behave and lay out artifacts exactly as before (FR-015).

🤖 Generated with Claude Code


Assisted-by: speckit-opsmill-auto unversioned
Assisted-by: speckit-opsmill-implement unversioned
Assisted-by: speckit-opsmill-extract unversioned
Assisted-by: opsmill-dev-commit 0.1.0
Assisted-by: opsmill-dev-pr 0.2.0


Summary by cubic

Adds backup and restore for Kubernetes deployments whose Neo4j Enterprise or PostgreSQL database runs outside the cluster. infrahub-backup create and restore previously could not reach those databases; they now use the vendor's backup client in a transient in-namespace Job and stream artifacts through the existing pipeline. Fully internal deployments are unchanged.

New Features

  • The transient Job runs the infrahub-backup image, which ships neo4j-admin and the PostgreSQL client, so no vendor image pull is needed; when the tool itself runs as a pod in the namespace, direct mode runs the tools in-process and creates no Job or Secret.
  • Credentials reuse the deployment's own Secret keys when explicit secretKeyRef references exist, falling back to a run-owned Secret otherwise.
  • Restoring an external Neo4j database stages artifacts in object storage and polls the database to confirm recovery; destructive external restores need separate authorization, external TLS is verified by default, and the Job's time-to-live removes the pod and its credential Secret after a killed run.
  • Adds operator documentation for prerequisites, configuration, troubleshooting, and direct-to-cloud Neo4j backups.

Bug Fixes

  • Database location resolution fails on Kubernetes query, permission, or workload-listing errors instead of assuming an external database; each database reports its location independently.
  • Restores gate only the databases their archive carries; Docker restores wait for freshly healthy databases before starting dependent services; external capture streams report truncation and timeout failures instead of retaining incomplete artifacts.
  • Credential reuse ignores envFrom-backed variables, and the PostgreSQL keyword/value connection form is parsed like libpq.
  • With --k8s-namespace or --project set, a refused or missing namespace fails the run with kubectl's message instead of falling through to a Docker Compose project.
  • Direct-mode runs use an absolute scratch path, cancel the child's process group, and bound copies by --external-db-timeout.

Written for commit 304756b. Summary will update on new commits.

View guided diff Turn on auto-fix

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Deploying infrahub-ops-cli with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9a61410
Status: ✅  Deploy successful!
Preview URL: https://4dab6500.infrahub-ops-cli.pages.dev
Branch Preview URL: https://007-external-database-backup.infrahub-ops-cli.pages.dev

View logs

Infrahub deployments on Kubernetes whose Neo4j Enterprise database,
PostgreSQL task-manager database, or both run outside the cluster had no
backup or recovery path through this toolset: every path executed the
vendor utility inside a database container, and there was none. This
supplies one — a short-lived Job in the namespace acting as the vendor's
own backup client — so the existing execution, streaming and copy
primitives work unchanged and a deployment whose databases are all
internal behaves exactly as before (FR-015).

Capture runs `neo4j-admin database backup --from=…` or `pg_dump` in that
workload and streams the artifact through the existing pipeline, so an
artifact taken externally is byte-compatible with an internal one and
restores by the same code. Where a database lives is settled at a gate
from positive evidence: a deployment query that fails is an error naming
the permission, never a silent switch to the external path, so a
permission problem cannot create workloads or pull images (FR-002).
Everything that would yield an unusable artifact is caught before data
moves — a utility older than the server, an external Community database,
missing workload permissions, a restore artifact the server cannot read.

Restore inverts the data path, because the server fetches the artifact
itself. The raw backup is staged in the configured object store, proven
readable by an independent client, seeded over Bolt, and then confirmed
online by polling: an accepted seed is not a completed one (FR-021). It
requires its own authorisation, distinct from --force and from enabling
scheduled restore (FR-009).

Nothing outlives the run. The workload carries its own deadline, its
credential Secret is garbage-collected with it, and a reaper removes
strays left by earlier runs (FR-011, FR-023). Connections are encrypted
and verified by default, with one operator opt-out (FR-029). Incomplete
captures are removed rather than retained, completeness is read from the
backup command's output rather than its exit status, and metadata
additions are additive so older releases keep restoring new artifacts
(FR-012, FR-016, FR-024).

Kubernetes resource ownership is decided in one place from label evidence
and declared release prefixes, replacing the substring matches that could
scale an unrelated workload to zero.

Spec, research, contracts and task ledger are archived under
specs/archive/007-external-database-backup/. The decisions are recorded
as ADRs 0008-0016, with operator documentation under docs/docs/backup/
and developer notes under dev/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@BeArchiTek
BeArchiTek force-pushed the 007-external-database-backup branch from bd4d482 to 3f27cdb Compare September 10, 2026 13:52
@BeArchiTek
BeArchiTek marked this pull request as ready for review September 14, 2026 08:59
@BeArchiTek
BeArchiTek requested a review from a team as a code owner September 14, 2026 08:59

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 99 files

Not reviewed (too large): specs/archive/007-external-database-backup/tasks.md (~441 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md">

<violation number="1" location="specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md:95">
P3: E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.</violation>
</file>

<file name="src/internal/app/backup_containers.go">

<violation number="1" location="src/internal/app/backup_containers.go:293">
P2: The quiesce and restart waits share one deadline across all services. If the first service consumes the window (a stuck pod terminating, or a slow image pull), every later service is checked with the deadline already passed and is reported as not down / not running without ever being polled. Compute the deadline per service (or subtract elapsed time before each service's loop) so each stopped/restarted workload gets the full window.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/internal/app/backup_neo4j.go Resolved
Comment thread src/internal/app/backup.go Resolved
Comment thread src/internal/app/app_config.go Resolved
Comment thread Makefile Resolved

### 4a. Architecture Soundness

**E6 🎯** — Pod resolution falls back to **substring matching on pod names**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At specs/archive/007-external-database-backup/critiques/critique-20260902-1330.md, line 95:

<comment>E6's technical premise does not match the code in the tree. It states pod resolution "falls back to substring matching on pod names (environment_kubernetes.go:305-313...)" and that the transient backup pod "would be reported as a replica of the database service" to GetAllPods/ServiceReplicas. In the current implementation environment_kubernetes.go:300-315 is CopyTo/CopyFrom (no substring logic), the resolver is a label-selector tier plus an ownership fallback (podsForServiceWith/claimedPodNamesWith), and getAllPodsWith explicitly excludes the transient workload via podListingArgsFor and deliberately keeps the transient pod's name free of any service-name substring (FR-028, transientObjectPrefix comments). The problem E6 describes is already solved, and its own recommendation is already implemented. As this critique is being merged as a technical record in specs/archive, the stale and incorrect citation will mislead readers about the current resolver.</comment>

<file context>
@@ -0,0 +1,297 @@
+
+### 4a. Architecture Soundness
+
+**E6 🎯** — Pod resolution falls back to **substring matching on pod names**
+(`environment_kubernetes.go:305-313`, and the same fallback in `getAllPodsWith`). The transient pod
+must contain the canonical service name for that fallback to find it — but the same property makes it
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3584a53. Kept the point-in-time finding and added a dated resolution note naming the current locations (podsForServiceWith, claimedPodNamesWith, parseLabelledPods applying withoutTransientPods for FR-028).

Comment thread src/internal/app/cli_test.go Resolved
@BeArchiTek
BeArchiTek force-pushed the 007-external-database-backup branch from 3f27cdb to 8df6a24 Compare September 14, 2026 21:53
BeArchiTek and others added 12 commits October 5, 2026 15:03
…ating a database

deploymentClaimsService reads Deployments and StatefulSets only once no
pod declares the service, which is exactly when they decide the answer: a
StatefulSet scaled to zero is internal only because its workload says so.
A failed workload listing was logged at debug level and skipped, so a
forbidden or unreachable listing returned "external" for a question that
was never answered. FR-002 names permission denied as a failure to query,
not as evidence.

listAllWorkloadsWith now returns the error, deploymentClaimsService and
locateServiceWith carry it into undeterminedLocation, and the RBAC hint
names deployments and statefulsets beside pods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
minio/minio is no longer published on Docker Hub or quay.io, so every
S3 e2e test errored in fixture setup with ImageNotFound. pgsty/minio is
a community build of the same server; the fixture command and health
endpoint are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slow external stream that kept making progress never tripped the idle
timeout, so the transient pod's activeDeadlineSeconds could end it. The
pod's removal ends kubectl's stdout like a clean finish, and kloset
discards the reader's Close error, so the truncated archive was
committed as a complete snapshot.

execReadCloser now waits for the producing command on EOF and returns
its failure from Read, which kloset records as a source error and the
snapshot commit refuses. The external Neo4j and PostgreSQL streams are
also held to the capture's bound as a whole, the full-bound slot the
capture workload's deadline already budgets for the artifact leaving
the pod, so the tool ends a slow transfer before the cluster does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RestoreBackup settled where each database lives from --exclude-taskmanager
alone, before the archive was read. An archive without the task-manager
database was then refused, or given a transient workload, for an external
PostgreSQL the restore never writes. The gate now runs after the metadata
is parsed, on taskManagerIncluded && !excludeTaskManager, and still before
the edition probe, the wipe and the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s in every restore path

e200950 moved RestoreBackup's gate after the archive metadata is read. The
Plakar restore and the S3 leg of restore --latest still gated the
task-manager database on --exclude-taskmanager alone, before the backup was
read, so a backup without that database was refused, or given a transient
workload, for an external PostgreSQL the restore never writes.

The Plakar group restore now gates once the group is chosen, on whether it
holds a postgres snapshot; the single-snapshot restore gates once the
snapshot's component is read, before the edition probe. The S3 leg gates
only Neo4j before the download, which every archive carries, and leaves the
task manager to RestoreBackup's gate after the metadata is read; the second
pass over Neo4j prepares nothing again. All gates still run before anything
is stopped or wiped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… way libpq does

statedKeywordFields split the keyword/value form on whitespace, so a quoted
value holding a space, such as sslrootcert='/etc/ssl/my certs/ca.crt', was
cut at the space, and whitespace around "=" lost the setting. Discovery then
recorded a truncated host or certificate path for the external task-manager
database.

The form is now parsed as libpq parses it: whitespace around "=", single-
quoted values, and backslash escapes for \' and \\. A string libpq would
refuse (an unterminated quote, a keyword without "=") is an error, and
discovery reads nothing from it rather than part of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cannot be located

environment detect resolved both databases through resolveDatabaseTargets,
which stops at the first location query that fails. One failed query hid the
other database from the report. The report now resolves each database on its
own through databaseTargetWith and logs a per-database warning; the run gate
still stops at the first unknown location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… query

The fake backend answered every cypher-shell call with success, so the
subtest exercised the parse of a successful answer and never the
assume-not-clustered branch it is named for. The fake can now fail
cypher-shell calls, and the subtest asserts the query was issued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Run golangci-lint v2.13.2 in CI, matching the dev-setup pin.
- Share the build tag between GO_TAGS and the lint target.
- Clear INFRAHUB_EXTERNAL_DB_INSECURE_TLS in the CLI test environment.
- Correct the spec directory path in plan.md, reconcile the open-task
  count in the implementation report, note where critique E6 was
  resolved, and align the backup artefact wording with FR-005.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c560786 made execReadCloser wait for the producing command when the
stream reaches EOF. Waiting closes the command's pipe, and kloset reads
once more after an EOF, so that read returned "file already closed".
kloset recorded it as a source error and every Plakar Neo4j snapshot was
refused, on Docker and Kubernetes alike.

Once the command has been waited for, Read now returns EOF. The new
test streams a real process through kloset, which the fake pipes in the
existing tests could not reproduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…paths

- restore --latest --s3 resolves database credentials before gating, so
  the external Neo4j target carries the default database and user
- --reset-deployment-id runs through the restore pod when Neo4j is external
- a single Plakar postgres-snapshot restore restarts every service it
  stopped, and gates only the database it writes
- the restore pod's deadline covers every bound the pod is held through
- docker IsRunning treats a service the compose file lacks as not running
- the Prefect chart's component=server label no longer claims a pod for
  infrahub-server
- resolveJobPod falls back to polling when kubectl predates --for=create
- drop statedPrefectSSLRootCert, which only tests called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread src/internal/app/backup_deployment.go Resolved
Comment thread src/internal/app/transient_workload.go Resolved
BeArchiTek and others added 4 commits October 5, 2026 16:25
…deadline

When Neo4j is external the deployment-ID reset runs through the restore
workload, under the probe bound and retried up to ten times. That worst case
(about 50 minutes at the defaults) was not in the pod's activeDeadlineSeconds,
and the 10-minute margin cannot absorb it, so the cluster could remove the pod
during the reset and fail a restore that had already completed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The polling fallback for kubectl without --for=create ran each listing under
the control bound, independently of the poll's deadline. A listing started just
before the deadline could return after it, stretching pod resolution past its
ten-minute bound and accepting a pod found late. Each listing now runs through
queryWithin with the time left, and none is issued once the deadline passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/api/config answers before infrahub-server can reach its database again
after the database pod reset between tests, so the next test's first SDK
call (the schema fetch) could get a 503. Wait for /api/schema as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docker

The Community streamed load resumes a Neo4j that then finishes its
shutdown and is restarted by its restart policy. `docker compose start
infrahub-server` checks the `service_healthy` dependency once and refuses
an exited database, so the post-restore start raced that restart and
failed with a bare "exit status 1".

All restore paths now start infrahub-server and task-worker through one
helper that, on Docker, waits (bounded, 5 min) for every database
container to be running and freshly healthy, starting it if it stays
exited. DockerBackend.Start and Stop include compose's output in their
errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

BeArchiTek and others added 2 commits October 5, 2026 16:56
The five-minute wait only checked its deadline between readings, so a hung
docker compose ps, docker inspect or docker compose start held it
indefinitely. Each call now runs with the time left before the deadline and
a call cut short returns a timeout error naming it. runCommandContext also
bounds the wait for a killed command's output pipe, which a child such as the
compose plugin keeps open after the docker CLI is killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its URL

/api/schema can answer while infrahub-server still cannot reach Neo4j
after the database pod reset between tests, so the next test's first
mutation failed with 'Unable to connect to the database'. Wait for a
GraphQL query that reads the database instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/internal/app/docker_service_ready_test.go">

<violation number="1" location="src/internal/app/docker_service_ready_test.go:170">
P3: These fake Docker scripts leave each `sleep 30` child running for 30 seconds: `CommandContext` kills only the shell, and `WaitDelay` abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

bin := t.TempDir()
// Not `exec sleep`: the sleep is a child that keeps the output pipe open
// after the killed shell, as the compose plugin does under the docker CLI.
script := "#!/bin/sh\nsleep 30\n"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These fake Docker scripts leave each sleep 30 child running for 30 seconds: CommandContext kills only the shell, and WaitDelay abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/internal/app/docker_service_ready_test.go, line 170:

<comment>These fake Docker scripts leave each `sleep 30` child running for 30 seconds: `CommandContext` kills only the shell, and `WaitDelay` abandons its inherited output pipe. Kill the child during test cleanup so repeated test runs do not accumulate orphan processes.</comment>

<file context>
@@ -160,3 +160,44 @@ func TestDockerBackendStartIncludesComposeOutput(t *testing.T) {
+	bin := t.TempDir()
+	// Not `exec sleep`: the sleep is a child that keeps the output pipe open
+	// after the killed shell, as the compose plugin does under the docker CLI.
+	script := "#!/bin/sh\nsleep 30\n"
+	if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(script), 0o755); err != nil {
+		t.Fatalf("writing fake docker = %v", err)
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point — with the process-group kill, the fake scripts' sleep 30 children now die with the group when the timeout fires, so no orphans remain after the tests, in d4bb6b5.

BeArchiTek and others added 4 commits October 13, 2026 17:23
…losed

The child's context was cancelled only after cmd.Wait returned, so a stream
that hit its idle timeout or deadline closed stdout and then waited on a
child blocked in database I/O, which never notices the closed pipe. Closing
the stream now cancels the child first. Children run in their own process
group, so cancelling also stops what an `sh -c` pipeline started, and
WaitDelay bounds the wait for their output pipes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
copyAcrossTransientWorkload returned before taking externalDBBound when the
target was a local workload, so a copy into or out of the scratch directory
on a slow or stalled filesystem could hold a backup or restore past the
bound every other external operation runs under. Local copies now take a
context and stop between reads once it is done, and the seam runs them
under the same bound, returning the timeout at the bound even when a copy
is stuck inside one read or write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The external suites are Python, so a change to them alone skipped the job that
runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A missing Secret key is fixed by restoring the key; restarting the pods
  does not help.
- Direct mode also needs a temporary directory with a plain path.
- FR-031's refusal of a development build applies only to runs that create a
  transient workload.
- ADR 0008: the external path adds write permissions to those the Kubernetes
  path already needs. ADR 0018: the cleanup guarantee depends on the Job, TTL
  and garbage-collection controllers.
- The changelog says only named Secret keys avoid the secrets permission.
- The knowledge doc: a failed pod-spec read is not remembered, and the image to
  mirror is the one --external-db-image selects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 existing issue remains and no new issues found across 10 files (changes from recent commits).

You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/ci.yml Resolved
@BeArchiTek
BeArchiTek requested a review from fatih-acar October 13, 2026 15:49
With --k8s-namespace set and the namespace check refused, detection fell
through to Docker. In CI this picked up another job's Compose project on the
runner, and the run started backing it up; a restore would have written to
it. An explicit target now pins its backend, and that backend's detection
failure is the run's failure (tasks.md T156).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/docs/reference/configuration.mdx Resolved
- e2e-tests-k8s needs go-lint directly, so a lint failure stops it.
- ADR 0008 names the scale resources as the only write kubectl scale makes.
- The configuration reference describes what happens when both --project and
  --k8s-namespace are set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Daniel Pardo and others added 3 commits November 6, 2026 08:36
Resolve the conflicts between the external database work and the fixes
released in v2.4.0 and v2.4.1:

- restore: add --neo4j-cluster-online-timeout to newRestoreCommand
- k8s: try infrahub/service first in serviceLabelKeys, then the Neo4j
  Helm chart label for `database`, so the Neo4j service does not resolve
  to CloudNativePG pods (f673202, 1e0e522)
- backup and restore: record workload replica counts before anything is
  stopped (9bbb849), on the PR's edition gate
- Neo4j edition: keep requireDeterminedEdition and add the probe error
  to it; remove main's detectBackupNeo4jEdition, which did the same
- restore: follow main's rule that an unknown edition stops the restore
  of an archive without a recorded edition (84dc735), and update the
  PR test that expected a fallback
- remove the duplicate runCommandSeparate and newTestKubernetesBackend
- e2e: use main's Garage S3 server and authenticated /api/schema wait,
  then the PR's database query wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alpine removed openjdk21-jdk and openjdk21-jmods 21.0.12_p8-r0 when it
published 21.0.13.0.3_p1-r0, so the image build stopped at the
java-runtime stage. Both e2e-tests-k8s jobs on the merge of main failed
there. scripts/check-image-tools.sh passes on the rebuilt image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Kubernetes case ran against one Neo4j server, so the cluster
restore flow and the external path against a cluster had no end-to-end
coverage. Two topologies now run a three-member Neo4j cluster, built the
way Infrahub's HA example builds it: three Neo4j Helm releases and a
headless service, with Infrahub connecting through the neo4j protocol.

- internal: the cluster runs in the Infrahub namespace with the
  infrahub/service=database label. A tarball round trip checks that the
  restore takes the cluster flow and that the database is online on
  every member afterwards.
- external: the cluster runs in its own namespace, so the tool takes the
  external path. One case checks what the capture records for the
  cluster address, and one runs the full round trip, where the restore
  seeds every member from the object store.

The new e2e-tests-k8s-neo4j-cluster job runs each topology on its own
runner with the enterprise chart. Both topologies passed locally on a
docker-driver vcluster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Daniel Pardo and others added 2 commits November 6, 2026 11:00
mypy cannot index Popen.args, which python-lint reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Address the review of the Neo4j cluster tests:

- Create the vcluster with three nodes when a session declares a cluster
  topology, and keep the chart's anti-affinity, so each member runs on a
  node of its own. The install fails when the members share a node.
- Stage external restores in the session's Garage container (s3_server),
  attached to the vcluster's Docker network, instead of a MinIO container
  started by CI.
- Build the external topology in the fixtures: the address, the
  credentials, the flags and the infrahub-backup image. The CI job only
  declares the topology and runs pytest.

Add quality checks:

- After a restore, read every member on its own and check that all hold
  the same graph.
- Fail a session that declares a topology and passes none of its cases.
- Dump the cluster namespaces' logs when a cluster case fails.
- Upload the JUnit results of the cluster jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…space

Add a case to the external cluster suite that runs infrahub-backup in a
pod in the Infrahub namespace, the way the Helm chart runs it. The run
gets no --external-db-image, uses the tools of its own image (direct
mode, ADR 0017) and creates no transient Job. Its service account holds
only the direct-mode permissions. The case checks the backup and the
restore logs, that every member is online with the same graph, that the
pre-backup data is back, and that no transient object is left.

Move the direct-mode log lines into conftest so both suites share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danipl
danipl added this pull request to stack #200 November 9, 2026 08:58
Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how
to back up and restore an external database with the infrahub-backup
Helm chart.

Add a section to the external database guide that lists the chart's
externalDatabase values and the environment variable each one sets,
the settings the chart has no value for (scratch size, a larger
temporary volume), the restore requirements on the S3 bucket, and one
values example for a backup and one for a restore. Link it from the
prerequisites of the Kubernetes backup and restore guides, and point
the cluster online timeout troubleshooting at the new chart value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/docs/backup/kubernetes-restore.mdx">

<violation number="1" location="docs/docs/backup/kubernetes-restore.mdx:29">
P2: The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

- Access to modify Helm values for your Infrahub deployment
- S3 credentials with read access to the backup bucket
- Network egress from the restore pod to the Kubernetes API server (see [Network access to the API server](./kubernetes-backup.mdx#network-access-to-the-api-server))
- If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/backup/kubernetes-restore.mdx, line 29:

<comment>The external-database condition is wrong: it says "outside the Infrahub namespace", but the external-databases guide defines external as a database that does not run in this deployment. A database in the same namespace that the deployment does not manage still needs the same preparation and chart values, so readers would skip required steps. Name the condition as "not managed by this Infrahub deployment" and say it applies even inside the namespace.</comment>

<file context>
@@ -26,6 +26,7 @@ Before restoring:
 - Access to modify Helm values for your Infrahub deployment
 - S3 credentials with read access to the backup bucket
 - Network egress from the restore pod to the Kubernetes API server (see [Network access to the API server](./kubernetes-backup.mdx#network-access-to-the-api-server))
+- If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket
 
 ## Step 1: Identify the backup to restore
</file context>
Suggested change
- If Neo4j or PostgreSQL runs outside the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket
- If Neo4j or PostgreSQL is not managed by this Infrahub deployment, even when it runs in the Infrahub namespace: the database is prepared as described in [Prepare an external database for backup and restore](./external-databases.mdx), and the chart values are set as described in [Configure the `infrahub-backup` Helm chart](./external-databases.mdx#configure-the-infrahub-backup-helm-chart). A restore of an external Neo4j database also needs S3 credentials that can write to the bucket

Daniel Pardo and others added 3 commits November 16, 2026 10:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…23 ships

Alpine removed postgresql18-client 18.6-r0 when it published 18.7-r0, so
the image build stopped at the apk add step. e2e-tests-k8s (enterprise)
on the merge of main failed there. scripts/check-image-tools.sh passes
on the rebuilt image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
```bash
infrahub-backup create \
--k8s-namespace infrahub \
--external-db-image registry.example.com/opsmill/infrahub-backup:2.3.0 \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it doesn't make sense to have that flag, because users in this case will already run infrahub-backup through their registry using the helm chart
We really need to have integration and testing in infrahub-helm before merging this PR


## Consequences

- Customer prerequisites that no earlier feature had: each Neo4j server needs a route to the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets keep it

@danipl
danipl merged commit 25f8858 into main Nov 16, 2026
18 checks passed
@danipl
danipl deleted the 007-external-database-backup branch November 16, 2026 10:50
danipl pushed a commit to opsmill/infrahub-helm that referenced this pull request Nov 16, 2026
…eased image [IFC-3553]

infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub
namespace, and the chart now uses it. Remove the build of an image from
opsmill/infrahub-backup#172: the Jobs run the image of the chart's
appVersion. Move the test from the manual marker to the backup marker, so
the e2e-infrahub-backup CI job runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
danipl added a commit to opsmill/infrahub-helm that referenced this pull request Nov 16, 2026
…ce (#125)

* feat(infrahub-backup): support databases outside the Infrahub namespace [IFC-3553]

Add externalDatabase values for a Neo4j or PostgreSQL database that runs
outside the Infrahub namespace. Each one sets the matching infrahub-backup
environment variable: allowRestore (restore containers only), insecureTLS,
timeout, scratchSize, neo4jAddress, neo4jBackupPort and postgresAddress.
scratchVolume mounts a volume at /scratch and sets TMPDIR to it.

A restore by restore.storage.s3.key now also sets INFRAHUB_S3_BUCKET,
INFRAHUB_S3_PREFIX and INFRAHUB_S3_ENDPOINT. A restore of an external Neo4j
database copies the archive there for the Neo4j servers to download.

Add restore.options.neo4jClusterOnlineTimeout, which passes
--neo4j-cluster-online-timeout (infrahub-backup 2.4.1 or later).

All new values are empty by default, so the rendered chart does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(infrahub-backup): back up and restore an external Neo4j through the chart [IFC-3553]

Add a manual e2e test that installs a Neo4j 2025.10.1 Enterprise server in
its own namespace and Infrahub with neo4j.enabled=false pointed at it. The
backup and restore Jobs of the chart must run the database tools in their own
container, create no Job and no Secret, write their temporary files to an
ephemeral PersistentVolumeClaim, and bring back the data from before the
backup.

The test needs an infrahub-backup image built from
opsmill/infrahub-backup#172, so it is marked manual until that change is
released. Move the Garage setup into deploy_garage so both backup tests use
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(infrahub-backup): use infrahub-backup 2.4.1 and document restore bucket access [IFC-3553]

Bump appVersion to 2.4.1. restore.options.neo4jClusterOnlineTimeout passes
--neo4j-cluster-online-timeout, which infrahub-backup 2.4.0 does not have,
so a restore with this value failed with the default image.

Document that a restore of an external Neo4j database writes the staged
artifact to restore.storage.s3.bucket, also with restore.storage.s3.key. The
credentials must be able to write to that bucket, and the Neo4j servers must
be able to read from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(infrahub-backup): remove externalDatabase.scratchVolume [IFC-3553]

The chart does not need its own value for the volume of the temporary files.
An operator who needs a larger volume can add it with extraVolumes and
extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now
says so. The manual e2e test no longer mounts a scratch volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(infrahub-backup): run the external Neo4j test without --external-db-image [IFC-3553]

The backup and restore Jobs run in the Infrahub namespace, so infrahub-backup
runs the database tools in its own container and creates no Job. It checks
the image of that Job only when it creates one, so the flag is not needed.
The test now runs the chart as an operator does, with no extra arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(infrahub-backup): set up the external database namespace in a fixture [IFC-3553]

Move the namespace and the Garage of the external Neo4j test into the
external_db_s3_k8s fixture, as s3_k8s does for the community deployment.
infrahub_external_neo4j_k8s now depends on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(infrahub-backup): remove externalDatabase.scratchSize [IFC-3553]

The chart no longer has a value for the free disk space of a capture of an
external database. An operator who needs it sets
INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025
server needs it, because the server does not report its store size.

The manual e2e test now sets the variable with extraEnv, in a values file
that also keeps the S3 CA settings: a --set on extraEnv would replace the
list of the CA values file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(infrahub-backup): use infrahub-backup 2.6.0 [IFC-3553]

Set appVersion to 2.6.0, the first infrahub-backup release that backs up and
restores a database outside the Infrahub namespace, which the
externalDatabase values need. Replace the 2.4.1 changelog fragment, which
infrahub-backup-1.6.0 already released, with one for 2.6.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(infrahub-backup): run the external Neo4j test in CI with the released image [IFC-3553]

infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub
namespace, and the chart now uses it. Remove the build of an image from
opsmill/infrahub-backup#172: the Jobs run the image of the chart's
appVersion. Move the test from the manual marker to the backup marker, so
the e2e-infrahub-backup CI job runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(infrahub-observability): record every trace in the observability e2e test

Infrahub 1.12 adds trace.sample_ratio, which records 10% of new traces by
default. The trace of the probe request of
test_grafana_can_query_infrahub_metrics_and_traces was then usually not
recorded, so Tempo had no trace to find and the test timed out. Set
INFRAHUB_TRACE_SAMPLE_RATIO=1 on the Infrahub server in the fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Daniel Pardo <daniel@opsmill.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changes/minor Release with new features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants