Repository navigation
feat(infrahub-backup): support databases outside the Infrahub namespace - #125
Conversation
…ce [IFC-3553] Add externalDatabase values for a Neo4j or PostgreSQL database that runs outside the Infrahub namespace. Each one sets the matching infrahub-backup environment variable: allowRestore (restore containers only), insecureTLS, timeout, scratchSize, neo4jAddress, neo4jBackupPort and postgresAddress. scratchVolume mounts a volume at /scratch and sets TMPDIR to it. A restore by restore.storage.s3.key now also sets INFRAHUB_S3_BUCKET, INFRAHUB_S3_PREFIX and INFRAHUB_S3_ENDPOINT. A restore of an external Neo4j database copies the archive there for the Neo4j servers to download. Add restore.options.neo4jClusterOnlineTimeout, which passes --neo4j-cluster-online-timeout (infrahub-backup 2.4.1 or later). All new values are empty by default, so the rendered chart does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the chart [IFC-3553] Add a manual e2e test that installs a Neo4j 2025.10.1 Enterprise server in its own namespace and Infrahub with neo4j.enabled=false pointed at it. The backup and restore Jobs of the chart must run the database tools in their own container, create no Job and no Secret, write their temporary files to an ephemeral PersistentVolumeClaim, and bring back the data from before the backup. The test needs an infrahub-backup image built from opsmill/infrahub-backup#172, so it is marked manual until that change is released. Move the Garage setup into deploy_garage so both backup tests use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bucket access [IFC-3553] Bump appVersion to 2.4.1. restore.options.neo4jClusterOnlineTimeout passes --neo4j-cluster-online-timeout, which infrahub-backup 2.4.0 does not have, so a restore with this value failed with the default image. Document that a restore of an external Neo4j database writes the staged artifact to restore.storage.s3.bucket, also with restore.storage.s3.key. The credentials must be able to write to that bucket, and the Neo4j servers must be able to read from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| repository, _, tag = image.rpartition(":") | ||
|
|
||
| subprocess.run(["kubectl", "--kubeconfig", kubeconfig, "create", "namespace", namespace], check=True) | ||
| s3 = await deploy_garage(kubeconfig, namespace) |
There was a problem hiding this comment.
why aren't we using it as a fixture here?
There was a problem hiding this comment.
No good reason. Moved in 5c7d5ca: the new external_db_s3_k8s fixture creates the namespace and deploys Garage, as s3_k8s does for the community deployment, and infrahub_external_neo4j_k8s depends on it.
| s3 = deployment["s3"] | ||
| chart = CHARTS_DIR / "infrahub-backup" | ||
| common = _common_sets(deployment) | ||
| image_arg = f"--external-db-image={deployment['backup_image']}" |
There was a problem hiding this comment.
We need to test without --external-db-image
There was a problem hiding this comment.
Valid point. The Jobs run in direct mode, and infrahub-backup checks the image only when it creates a Job, so the flag was not needed. Removed in 218c007: the test now runs the chart with no extra arguments, as an operator does. The manual e2e test passes without the flag (1 passed in 5m47s).
| volumeMounts: | ||
| - name: backup-storage | ||
| mountPath: {{ .Values.backup.storage.path }} | ||
| {{- include "infrahub-backup.scratchVolumeMount" . | nindent 16 }} |
There was a problem hiding this comment.
I don't think the scratch volume is required for this PR, we can add it later if really required
There was a problem hiding this comment.
Agreed, removed in cc119cb. An operator who needs a larger volume for the temporary files can add it with extraVolumes and extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now says this. The manual e2e test no longer mounts a scratch volume, and it still passes.
There was a problem hiding this comment.
I think you can remove all mentions of "scratch" in this PR
There was a problem hiding this comment.
Done in 5993a8a: the chart no longer has externalDatabase.scratchSize, and the chart has no other mention of "scratch". An operator who needs the size sets INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025 server needs it: the server does not report its store size, and infrahub-backup stops with an error when it has neither. The manual e2e test now sets the variable with extraEnv, so "scratch" is only left there, as the name of the variable. The test passes (1 passed in 6m09s).
…-3553] The chart does not need its own value for the volume of the temporary files. An operator who needs a larger volume can add it with extraVolumes and extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now says so. The manual e2e test no longer mounts a scratch volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-db-image [IFC-3553] The backup and restore Jobs run in the Infrahub namespace, so infrahub-backup runs the database tools in its own container and creates no Job. It checks the image of that Job only when it creates one, so the flag is not needed. The test now runs the chart as an operator does, with no extra arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xture [IFC-3553] Move the namespace and the Garage of the external Neo4j test into the external_db_s3_k8s fixture, as s3_k8s does for the community deployment. infrahub_external_neo4j_k8s now depends on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fatih-acar
left a comment
There was a problem hiding this comment.
Let's add some docs on how to use this feature using helm, in the infrahub-backup repo
| EXTERNAL_NEO4J_AUTH_SECRET = "neo4j-auth" | ||
|
|
||
|
|
||
| def build_infrahub_backup_image(workdir: Path) -> str: |
There was a problem hiding this comment.
Let's remove this once infrahub-backup is released with this new feature
There was a problem hiding this comment.
Agreed. The follow-up PR that bumps appVersion to the infrahub-backup release with this feature removes the image build and moves the test to the backup marker, so CI runs it. No change in this PR.
There was a problem hiding this comment.
Update: infrahub-backup 2.6.0 is released, so this PR does it instead of a follow-up. 9fc97d5 removes the image build and moves the test to the backup marker. CI ran it in e2e-infrahub-backup and it passed.
…553] The chart no longer has a value for the free disk space of a capture of an external database. An operator who needs it sets INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025 server needs it, because the server does not report its store size. The manual e2e test now sets the variable with extraEnv, in a values file that also keeps the S3 CA settings: a --set on extraEnv would replace the list of the CA values file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how to back up and restore an external database with the infrahub-backup Helm chart. Add a section to the external database guide that lists the chart's externalDatabase values and the environment variable each one sets, the settings the chart has no value for (scratch size, a larger temporary volume), the restore requirements on the S3 bucket, and one values example for a backup and one for a restore. Link it from the prerequisites of the Kubernetes backup and restore guides, and point the cluster online timeout troubleshooting at the new chart value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| @@ -0,0 +1 @@ | |||
| Bumped the `infrahub-backup` image to `2.4.1`, which `restore.options.neo4jClusterOnlineTimeout` needs. See the [infrahub-backup 2.4.1 release notes](https://github.com/opsmill/infrahub-backup/releases/tag/v2.4.1). | |||
There was a problem hiding this comment.
Update this changelog after having released a newer infrahub-backup
There was a problem hiding this comment.
Done in a70bd88: the image is now 2.6.0, and +image-2.6.0.changed.md replaces the 2.4.1 fragment.
It links the 2.5.0 and 2.6.0 release notes.
* feat: back up and restore external databases on Kubernetes Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This supplies one — a short-lived Job in the namespace acting as the vendor's own backup client — so the existing execution, streaming and copy primitives work unchanged and a deployment whose databases are all internal behaves exactly as before (FR-015). Capture runs `neo4j-admin database backup --from=…` or `pg_dump` in that workload and streams the artifact through the existing pipeline, so an artifact taken externally is byte-compatible with an internal one and restores by the same code. Where a database lives is settled at a gate from positive evidence: a deployment query that fails is an error naming the permission, never a silent switch to the external path, so a permission problem cannot create workloads or pull images (FR-002). Everything that would yield an unusable artifact is caught before data moves — a utility older than the server, an external Community database, missing workload permissions, a restore artifact the server cannot read. Restore inverts the data path, because the server fetches the artifact itself. The raw backup is staged in the configured object store, proven readable by an independent client, seeded over Bolt, and then confirmed online by polling: an accepted seed is not a completed one (FR-021). It requires its own authorisation, distinct from --force and from enabling scheduled restore (FR-009). Nothing outlives the run. The workload carries its own deadline, its credential Secret is garbage-collected with it, and a reaper removes strays left by earlier runs (FR-011, FR-023). Connections are encrypted and verified by default, with one operator opt-out (FR-029). Incomplete captures are removed rather than retained, completeness is read from the backup command's output rather than its exit status, and metadata additions are additive so older releases keep restoring new artifacts (FR-012, FR-016, FR-024). Kubernetes resource ownership is decided in one place from label evidence and declared release prefixes, replacing the substring matches that could scale an unrelated workload to zero. Spec, research, contracts and task ledger are archived under specs/archive/007-external-database-backup/. The decisions are recorded as ADRs 0008-0016, with operator documentation under docs/docs/backup/ and developer notes under dev/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(external): stop the run when workloads cannot be listed while locating a database deploymentClaimsService reads Deployments and StatefulSets only once no pod declares the service, which is exactly when they decide the answer: a StatefulSet scaled to zero is internal only because its workload says so. A failed workload listing was logged at debug level and skipped, so a forbidden or unreachable listing returned "external" for a question that was never answered. FR-002 names permission denied as a failure to query, not as evidence. listAllWorkloadsWith now returns the error, deploymentClaimsService and locateServiceWith carry it into undeterminedLocation, and the RBAC hint names deployments and statefulsets beside pods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): pull MinIO from pgsty/minio minio/minio is no longer published on Docker Hub or quay.io, so every S3 e2e test errored in fixture setup with ImageNotFound. pgsty/minio is a community build of the same server; the fixture command and health endpoint are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): bound external capture streams and fail a truncated one A slow external stream that kept making progress never tripped the idle timeout, so the transient pod's activeDeadlineSeconds could end it. The pod's removal ends kubectl's stdout like a clean finish, and kloset discards the reader's Close error, so the truncated archive was committed as a complete snapshot. execReadCloser now waits for the producing command on EOF and returns its failure from Read, which kloset records as a source error and the snapshot commit refuses. The external Neo4j and PostgreSQL streams are also held to the capture's bound as a whole, the full-bound slot the capture workload's deadline already budgets for the artifact leaving the pod, so the tool ends a slow transfer before the cluster does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): gate a restore on the databases its archive carries RestoreBackup settled where each database lives from --exclude-taskmanager alone, before the archive was read. An archive without the task-manager database was then refused, or given a transient workload, for an external PostgreSQL the restore never writes. The gate now runs after the metadata is parsed, on taskManagerIncluded && !excludeTaskManager, and still before the edition probe, the wipe and the stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(restore): gate the task-manager database on the archive's contents in every restore path e200950 moved RestoreBackup's gate after the archive metadata is read. The Plakar restore and the S3 leg of restore --latest still gated the task-manager database on --exclude-taskmanager alone, before the backup was read, so a backup without that database was refused, or given a transient workload, for an external PostgreSQL the restore never writes. The Plakar group restore now gates once the group is chosen, on whether it holds a postgres snapshot; the single-snapshot restore gates once the snapshot's component is read, before the edition probe. The S3 leg gates only Neo4j before the download, which every archive carries, and leaves the task manager to RestoreBackup's gate after the metadata is read; the second pass over Neo4j prepares nothing again. All gates still run before anything is stopped or wiped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): parse the keyword/value PostgreSQL connection form the way libpq does statedKeywordFields split the keyword/value form on whitespace, so a quoted value holding a space, such as sslrootcert='/etc/ssl/my certs/ca.crt', was cut at the space, and whitespace around "=" lost the setting. Discovery then recorded a truncated host or certificate path for the external task-manager database. The form is now parsed as libpq parses it: whitespace around "=", single- quoted values, and backslash escapes for \' and \\. A string libpq would refuse (an unterminated quote, a keyword without "=") is an error, and discovery reads nothing from it rather than part of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(environment): report each database's location even when the other cannot be located environment detect resolved both databases through resolveDatabaseTargets, which stops at the first location query that fails. One failed query hid the other database from the report. The report now resolves each database on its own through databaseTargetWith and logs a per-database warning; the run gate still stops at the first unknown location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(neo4j): fail the cluster query in the subtest named for a failed query The fake backend answered every cypher-shell call with success, so the subtest exercised the parse of a successful answer and never the assume-not-clustered branch it is named for. The fake can now fail cypher-shell calls, and the subtest asserts the query was issued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(review): address review nitpicks - Run golangci-lint v2.13.2 in CI, matching the dev-setup pin. - Share the build tag between GO_TAGS and the lint target. - Clear INFRAHUB_EXTERNAL_DB_INSECURE_TLS in the CLI test environment. - Correct the spec directory path in plan.md, reconcile the open-task count in the implementation report, note where critique E6 was resolved, and align the backup artefact wording with FR-005. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plakar): keep a completed stream read past its EOF c560786 made execReadCloser wait for the producing command when the stream reaches EOF. Waiting closes the command's pipe, and kloset reads once more after an EOF, so that read returned "file already closed". kloset recorded it as a source error and every Plakar Neo4j snapshot was refused, on Docker and Kubernetes alike. Once the command has been waited for, Read now returns EOF. The new test streams a real process through kloset, which the fake pipes in the existing tests could not reproduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): address code-review findings on the external-database paths - restore --latest --s3 resolves database credentials before gating, so the external Neo4j target carries the default database and user - --reset-deployment-id runs through the restore pod when Neo4j is external - a single Plakar postgres-snapshot restore restarts every service it stopped, and gates only the database it writes - the restore pod's deadline covers every bound the pod is held through - docker IsRunning treats a service the compose file lacks as not running - the Prefect chart's component=server label no longer claims a pod for infrahub-server - resolveJobPod falls back to polling when kubectl predates --for=create - drop statedPrefectSSLRootCert, which only tests called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): count the deployment-ID reset in the restore workload deadline When Neo4j is external the deployment-ID reset runs through the restore workload, under the probe bound and retried up to ten times. That worst case (about 50 minutes at the defaults) was not in the pod's activeDeadlineSeconds, and the 10-minute margin cannot absorb it, so the cluster could remove the pod during the reset and fail a restore that had already completed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): bound the job-pod poll listings by the remaining wait The polling fallback for kubectl without --for=create ran each listing under the control bound, independently of the poll's deadline. A listing started just before the deadline could return after it, stretching pod resolution past its ten-minute bound and accepting a pod found late. Each listing now runs through queryWithin with the time left, and none is issued once the deadline passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): wait for the schema before handing a k8s test its URL /api/config answers before infrahub-server can reach its database again after the database pod reset between tests, so the next test's first SDK call (the schema fetch) could get a 503. Wait for /api/schema as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(restore): wait for a healthy database before starting the app on docker The Community streamed load resumes a Neo4j that then finishes its shutdown and is restarted by its restart policy. `docker compose start infrahub-server` checks the `service_healthy` dependency once and refuses an exited database, so the post-restore start raced that restart and failed with a bare "exit status 1". All restore paths now start infrahub-server and task-worker through one helper that, on Docker, waits (bounded, 5 min) for every database container to be running and freshly healthy, starting it if it stays exited. DockerBackend.Start and Stop include compose's output in their errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(restore): bound each docker call of the database readiness wait The five-minute wait only checked its deadline between readings, so a hung docker compose ps, docker inspect or docker compose start held it indefinitely. Each call now runs with the time left before the deadline and a call cut short returns a timeout error naming it. runCommandContext also bounds the wait for a killed command's output pipe, which a child such as the compose plugin keeps open after the docker CLI is killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): wait for a database-backed query before handing a k8s test its URL /api/schema can answer while infrahub-server still cannot reach Neo4j after the database pod reset between tests, so the next test's first mutation failed with 'Unable to connect to the database'. Wait for a GraphQL query that reads the database instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(restore): kill the whole process group when a readiness docker call times out A timeout killed only the docker CLI, so the compose plugin it runs as a child could carry on with `docker compose start` after the wait had reported it as timed out. The readiness wait's docker calls now run in a process group of their own, and the timeout kills the group. Windows keeps the single-process kill. The fake docker scripts in the tests spawn `sleep 30` children; they now die with the group instead of lingering for 30 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): retry Infrahub calls while it cannot reach its database After the database pod is replaced, one successful read does not mean every infrahub-server worker has reconnected: a later mutation still got 'Unable to connect to the database'. The readiness wait now needs three reads in a row, and the seed, verify and modify helpers retry that specific 503, which means the request did nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014fXAgGcEWm8mNkvxQVk538 * test(procgroup): count a zombie child as stopped kill(pid, 0) still finds a killed child until whoever adopted it reaps it, so the test could fail although the process-group kill worked. Ask ps for the state and treat a gone or zombie process as not running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014fXAgGcEWm8mNkvxQVk538 * feat(external): reference the deployment's own Secret for database credentials When the username and password a run holds were both read from the deployment component's environment, and that component's running pod declares the variables that held them as Secret keys (env secretKeyRef, or a single matching envFrom secretRef), the transient Job now maps those keys onto NEO4J_USERNAME/NEO4J_PASSWORD or PGUSER/PGPASSWORD with secretKeyRef. No Secret is created, so no secrets verbs are needed. The pod spec is read with a references-only jsonpath (pods get, already granted), so an inline value never reaches the tool. PostgreSQL reuse needs Secret-backed variables beside the Prefect URL that held the same user and password, as prefect-server renders with externalDatabase.usernameSecretKey/passwordSecretKey; a URL-only deployment falls back. Operator-supplied credentials, inline values, ConfigMaps, ambiguous envFrom sources and unreadable pods fall back to the Job-owned Secret unchanged. The stray reaper now warns instead of failing when listing secrets is forbidden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(adr): state that only an external database needs the new permissions ADR 0008 and 0009 listed the namespace grants first and the external-only condition last, so they read as a requirement on every deployment. They now lead with the condition, match the Job-based permissions the code needs, and mark the secrets verbs as needed only when the deployment's own Secret keys cannot be referenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(adr): record the tools-in-image and Job-workload decisions ADR 0017 moves the external-database tools into the infrahub-backup image, as asked in review: the transient Job runs that image, and a tool already running in the namespace runs them directly. ADR 0018 records the move from a bare Pod to a Job, whose time-to-live lets the cluster remove the pod and its credential Secret after a killed run. ADR 0008 and 0009 are marked as partly superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): add the tools-in-image amendment (FR-030 to FR-033) Requirements, plan section and task phase 6 for ADR 0017, asked for in review of PR #172. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(image): ship neo4j-admin, cypher-shell and the PostgreSQL 18 client The runtime image now carries the database tools the external-database path needs (FR-030, ADR 0017): - neo4j-admin and cypher-shell, with their jars and default conf, copied from neo4j:2025.10.1-enterprise into /opt/neo4j (NEO4J_HOME); NEO4J_CONF and NEO4J_LOGS point at directories the infrahub user owns - Alpine's OpenJDK 21 (openjdk21-jre-headless=21.0.12_p8-r0) - postgresql18-client=18.6-r0 (pg_dump, pg_restore, psql) The base moves from alpine:3.21 to alpine:3.23: 3.21 and 3.22 package no PostgreSQL 18 client. The Neo4j tools run on Alpine (musl), so the base stays Alpine. The image still runs as infrahub (uid 1000). scripts/check-image-tools.sh and `make docker-check-tools` build the image and run each tool's --version as the image's user, failing when a tool is missing or the image runs as root. Local run, 2026-10-12T10:28Z, docker 29.4.0 linux/arm64: Image user: infrahub (uid 1000) ok neo4j-admin: 2025.10.1 ok cypher-shell: Cypher-Shell 2025.10.1 ok pg_dump: pg_dump (PostgreSQL) 18.6 ok pg_restore: pg_restore (PostgreSQL) 18.6 ok psql: psql (PostgreSQL) 18.6 Image size: 360619641 bytes (343 MiB) All database tools present in infrahub-backup:check-tools Image size (docker image inspect .Size, linux/arm64): before: 34097838 bytes (33 MiB) after: 360619641 bytes (343 MiB) linux/amd64 builds too (364076638 bytes) and its tools print the same versions under emulation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(external): run the infrahub-backup image in the transient workload The transient Job now runs this tool's own image, which carries neo4j-admin, cypher-shell and the PostgreSQL 18 client (ADR 0017), in place of the neo4j and postgres vendor images (T164-T168, FR-031, FR-033). - One --external-db-image flag replaces --external-db-image-neo4j and --external-db-image-postgres. It stays unbound from the environment (FR-027). The old flags were never released, so they are removed. - The default is registry.opsmill.io/opsmill/infrahub-backup:<version>, with the leading "v" of the release tag stripped, as the release workflow pushes it. A development build (empty version, commit hash, git describe output, build metadata) has no default and is refused in externalCaptureOps, before the stray reaper and before any service is stopped, by a message naming the flag. - runAsUser, runAsGroup and fsGroup are 1000 for every role and both databases. The image declares its user by name, which the kubelet cannot check runAsNonRoot against, so the uid stays explicit. - Docs, contract and knowledge notes describe one image to mirror. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(external): run the database tools in-process when infrahub-backup runs in the namespace Direct mode (FR-032, ADR 0017, tasks T169 to T173). When the service-account namespace file names the target namespace and every tool a database needs is on PATH, externalCaptureOps returns operations served by a local workload: a run-specific scratch directory under os.TempDir. No Job or Secret is created, no run identifier is minted and the reaper does not run, so no jobs or secrets permission is exercised. The decision follows the collect refusal and precedes the image refusal, is taken once per database and is logged at info. The capture and restore sequences do not fork. Their execution target becomes local:<role>-<service>-<n>, and the seams they already call (execBounded, ExecStreamPipe, copyAcrossTransientWorkload, and the new refuseUnfittableRestoreDump) serve it in this process. The Job's /scratch mount path is translated onto the scratch directory in argv, environment values and copy paths, so the CA material lands there too. Credentials reach the child only through its environment (FR-014). Scratch fit is checked against statfs free space and the directory is removed on every exit path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): run the external suites on the image under test The Kubernetes e2e job builds infrahub-backup:e2e-<sha> and exports INFRAHUB_TESTING_EXTERNAL_DB_IMAGE. The external_db_image fixture loads it into the docker-driver vcluster with ctr import, and every external case passes it with --external-db-image, which is not bound to the environment (FR-027). The version-gate cases build an image from the one under test whose neo4j-admin reports 4.0.0 or 2099.12.0, so both directions run against the live server. The FR-023 case asserts the Secret is owned by the Job and deletes the Job (ADR 0018); transient_objects lists Jobs. A new case runs infrahub-backup as a Job in the namespace under the direct-mode permissions and asserts a capture with no transient workload (FR-032). Writing its Role showed the documented permissions missed list on deployments/statefulsets and patch on their scale resources; the docs are corrected. The external suites still skip until T072 provisions the topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): assert the cluster-side cleanup guarantee after a killed run A killed tool cannot delete its transient objects, and per ADR 0018 the cluster removes them only after the Job's deadline and time-to-live. The case now asserts what makes that happen: the Job carries a deadline and a time-to-live, the pod belongs to the Job and restates the deadline, and any credential Secret is owned by the Job. It then deletes the Job and waits for the garbage collector to remove the pod and the Secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): tighten scratch translation, child environment and published-version check - translate reads both boundaries of a /scratch occurrence in the original string, so /data/scratch/scratch is no longer rewritten. - Direct-mode children no longer inherit PG*, NEO4J_* (other than the image's NEO4J_HOME, NEO4J_CONF, NEO4J_LOGS) or the JVM option variables from this process, matching what the Job's container receives. - A git describe suffix after a pre-release (v2.4.0-rc.1-3-gabc) and a -dirty marker anywhere are refused as a published version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(image): trim the Java runtime and unused Neo4j jars The image drops from 343.9 MiB to 208.7 MiB compressed (arm64). The Java runtime is a jlink runtime with the modules the Neo4j tools use, and the jars neo4j-admin and cypher-shell never load are removed: the browser, the Cypher planner and runtimes, Scala, Parquet/Arrow, Jetty/Jersey/Prometheus/ JTS, and macOS/Windows native libraries. Validated on arm64 against a real Neo4j 2025.10.1 Enterprise and PostgreSQL 18: remote backup (plain and with a backup SSL policy), backup inspect, restore, consistency check, cypher-shell over plain, self-signed and verified TLS Bolt, pg_dump and pg_restore. amd64 could not run here, so CI now runs scripts/check-image-tools.sh on the amd64 runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(image): drop the cloud storage client libraries from the Neo4j tools The tool only ever passes neo4j-admin a local --to-path, and an S3 seed restore is fetched by the Neo4j server itself, so the S3, Azure and Google Cloud client jars are never loaded. Removing them takes the image from 208.7 MiB to 156.4 MiB compressed (arm64). Same validation as d1b3c2f; no class is loaded from a removed jar. The direct-to-cloud procedure page now says the image's neo4j-admin cannot write to cloud storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): report the phase 6 implementation run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): stage each restore seed in a folder of its own A Neo4j server seeding from a .backup URI reads the other artifacts beside it as one backup chain and refuses the seed when they do not form one. Every restore staged into one seed/ folder, and a failed restore keeps its staged copy on purpose, so that copy made every later restore into the same bucket fail with "not part of a valid backup chain". Observed on 2025.10.1 against MinIO: the same artifact seeded alone and failed next to another. The seed now goes to seed/<artifact name>/. Restoring the same archive twice writes the same object again. The upload failure also states that nothing was stopped, as the read-back failure already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(kubernetes): report why the namespace check failed With --k8s-namespace set, a refused or missing namespace was reported as "exit status 1". The error now carries kubectl's message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): run the external suites against a real external topology First run of the external suites (tasks.md T072, T175, T176). Corrections: - INFRAHUB_TESTING_EXTERNAL_INSECURE_TLS and _SCRATCH_SIZE add the flags every external run needs: the tool verifies certificates by default and always encrypts PostgreSQL, and Neo4j 2025.x reports no store size (T158) - the both-external overlay sets prefect-server.secret, without which the chart does not render - the post-run check waits for the cluster to remove the Job's pod - permission sets include workload listing and pod exec - the undiscoverable-endpoint case waits for infrahub-server's pod to go - source_roles is expected for Neo4j only; the component is task-manager-db Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the external-database suites against containers outside the cluster The enterprise leg of e2e-tests-k8s starts Neo4j Enterprise, PostgreSQL and MinIO on one Docker network, with a self-signed certificate on both servers, and runs the external Neo4j and both-external sessions (tasks.md T072). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): record the first external end-to-end run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): stop waiting when the seeded database goes offline with a reason Neo4j 2025.10.1 reports a rejected seed as offline with a statusMessage that does not contain "Unable to start database", so the restore waited the whole --external-db-timeout, two hours by default. An offline state with a message now ends the wait; offline with no message still waits, because a database being replaced can pass through it (tasks.md T178). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): one unreachable endpoint of several still captures the database neo4j-admin 2025.10.1 exits 0 and writes a backup of the whole database when one of two --from servers does not resolve, and FR-012 asks that every requested database be captured. The case now asserts the archive is kept, recorded as complete, and that the run warns about the endpoint it could not reach (tasks.md T177). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): record the T177 and T178 decisions and the TLS issue Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): T072 passes in CI; record the external end-to-end run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(spec-007): bring task status and test comments in line with the CI run The external suites now run in the enterprise leg of e2e-tests-k8s, so the skip reason, module docstrings and task entries no longer say T072 is open. T054 to T057 are ticked; T064 stays open because its Community case needs a server this topology does not provide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): do not reference a Secret key reached only through envFrom An envFrom source names no keys, and the kubelet skips a key the Secret lacks, so a variable whose name matches a source's prefix (an empty prefix matches every name) may come from the image instead. The generated secretKeyRef is mandatory, so a missing key held the transient pod pending and blocked the run. Checking the key would need get on secrets, which the reference path exists to avoid, so such a variable now falls back to the Job-owned Secret. Explicit env secretKeyRefs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): record PostgreSQL provenance from the values the read applied A partial operator connection URL (a user without a password) left the credential check unsatisfied, so the task-manager read replaced both values with the deployment's. The guard tested the values in place before that read, so the deployment-sourced pair was never recorded and the run fell back to creating a Job-owned Secret. Record provenance when the deployment URL itself carried both the user and the password, which applyPrefectConnection then applies. A URL without a password still leaves the operator's in place and records nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): make the direct-mode scratch path absolute A relative TMPDIR gave a relative scratch path, which is both the child's working directory and a path in its arguments, so once the child changed into it every scratch argument resolved beneath itself. Resolve the parent to an absolute path before creating the scratch directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): stop a direct-mode stream's child when the stream is closed The child's context was cancelled only after cmd.Wait returned, so a stream that hit its idle timeout or deadline closed stdout and then waited on a child blocked in database I/O, which never notices the closed pipe. Closing the stream now cancels the child first. Children run in their own process group, so cancelling also stops what an `sh -c` pipeline started, and WaitDelay bounds the wait for their output pipes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(external): bound direct-mode copies by --external-db-timeout copyAcrossTransientWorkload returned before taking externalDBBound when the target was a local workload, so a copy into or out of the scratch directory on a slow or stalled filesystem could hold a backup or restore past the bound every other external operation runs under. Local copies now take a context and stop between reads once it is done, and the seam runs them under the same bound, returning the timeout at the bound even when a copy is stuck inside one read or write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the Kubernetes end-to-end job when only Python changed The external suites are Python, so a change to them alone skipped the job that runs them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(external): correct statements the review found inaccurate - A missing Secret key is fixed by restoring the key; restarting the pods does not help. - Direct mode also needs a temporary directory with a plain path. - FR-031's refusal of a development build applies only to runs that create a transient workload. - ADR 0008: the external path adds write permissions to those the Kubernetes path already needs. ADR 0018: the cleanup guarantee depends on the Job, TTL and garbage-collection controllers. - The changelog says only named Secret keys avoid the secrets permission. - The knowledge doc: a failed pod-spec read is not remembered, and the image to mirror is the one --external-db-image selects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: act only on the deployment --k8s-namespace or --project names With --k8s-namespace set and the namespace check refused, detection fell through to Docker. In CI this picked up another job's Compose project on the runner, and the run started backing it up; a restore would have written to it. An explicit target now pins its backend, and that backend's detection failure is the run's failure (tasks.md T156). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(review): address the second wave of review comments - e2e-tests-k8s needs go-lint directly, so a lint failure stops it. - ADR 0008 names the scale resources as the only write kubectl scale makes. - The configuration reference describes what happens when both --project and --k8s-namespace are set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(image): bump the OpenJDK pin to the version Alpine 3.23 ships Alpine removed openjdk21-jdk and openjdk21-jmods 21.0.12_p8-r0 when it published 21.0.13.0.3_p1-r0, so the image build stopped at the java-runtime stage. Both e2e-tests-k8s jobs on the merge of main failed there. scripts/check-image-tools.sh passes on the rebuilt image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): run backup and restore against Neo4j clusters Every Kubernetes case ran against one Neo4j server, so the cluster restore flow and the external path against a cluster had no end-to-end coverage. Two topologies now run a three-member Neo4j cluster, built the way Infrahub's HA example builds it: three Neo4j Helm releases and a headless service, with Infrahub connecting through the neo4j protocol. - internal: the cluster runs in the Infrahub namespace with the infrahub/service=database label. A tarball round trip checks that the restore takes the cluster flow and that the database is online on every member afterwards. - external: the cluster runs in its own namespace, so the tool takes the external path. One case checks what the capture records for the cluster address, and one runs the full round trip, where the restore seeds every member from the object store. The new e2e-tests-k8s-neo4j-cluster job runs each topology on its own runner with the enterprise chart. Both topologies passed locally on a docker-driver vcluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): keep the release name with each Neo4j install process mypy cannot index Popen.args, which python-lint reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): build the Neo4j cluster topologies in pytest on three nodes Address the review of the Neo4j cluster tests: - Create the vcluster with three nodes when a session declares a cluster topology, and keep the chart's anti-affinity, so each member runs on a node of its own. The install fails when the members share a node. - Stage external restores in the session's Garage container (s3_server), attached to the vcluster's Docker network, instead of a MinIO container started by CI. - Build the external topology in the fixtures: the address, the credentials, the flags and the infrahub-backup image. The CI job only declares the topology and runs pytest. Add quality checks: - After a restore, read every member on its own and check that all hold the same graph. - Fail a session that declares a topology and passes none of its cases. - Dump the cluster namespaces' logs when a cluster case fails. - Upload the JUnit results of the cluster jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): back up and restore the Neo4j cluster from inside the namespace Add a case to the external cluster suite that runs infrahub-backup in a pod in the Infrahub namespace, the way the Helm chart runs it. The run gets no --external-db-image, uses the tools of its own image (direct mode, ADR 0017) and creates no transient Job. Its service account holds only the direct-mode permissions. The case checks the backup and the restore logs, that every member is online with the same graph, that the pre-backup data is back, and that no transient object is left. Move the direct-mode log lines into conftest so both suites share them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(backup): configure the Helm chart for an external database Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how to back up and restore an external database with the infrahub-backup Helm chart. Add a section to the external database guide that lists the chart's externalDatabase values and the environment variable each one sets, the settings the chart has no value for (scratch size, a larger temporary volume), the restore requirements on the S3 bucket, and one values example for a backup and one for a restore. Link it from the prerequisites of the Kubernetes backup and restore guides, and point the cluster online timeout troubleshooting at the new chart value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(image): bump the PostgreSQL client pin to the version Alpine 3.23 ships Alpine removed postgresql18-client 18.6-r0 when it published 18.7-r0, so the image build stopped at the apk add step. e2e-tests-k8s (enterprise) on the merge of main failed there. scripts/check-image-tools.sh passes on the rebuilt image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Daniel Pardo <daniel@opsmill.com>
…external-databases-ifc-3553 # Conflicts: # tests/README.md
Set appVersion to 2.6.0, the first infrahub-backup release that backs up and restores a database outside the Infrahub namespace, which the externalDatabase values need. Replace the 2.4.1 changelog fragment, which infrahub-backup-1.6.0 already released, with one for 2.6.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eased image [IFC-3553] infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub namespace, and the chart now uses it. Remove the build of an image from opsmill/infrahub-backup#172: the Jobs run the image of the chart's appVersion. Move the test from the manual marker to the backup marker, so the e2e-infrahub-backup CI job runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… e2e test Infrahub 1.12 adds trace.sample_ratio, which records 10% of new traces by default. The trace of the probe request of test_grafana_can_query_infrahub_metrics_and_traces was then usually not recorded, so Tempo had no trace to find and the test timed out. Set INFRAHUB_TRACE_SAMPLE_RATIO=1 on the Infrahub server in the fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| # Infrahub 1.12 records 10% of new traces by default; record all of | ||
| # them, so the trace of the probe request reaches Tempo. | ||
| "infrahubServer.infrahubServer.env.INFRAHUB_TRACE_SAMPLE_RATIO": "1", |
There was a problem hiding this comment.
is it related to the current PR?
There was a problem hiding this comment.
Not directly. Infrahub 1.12 records only 10% of traces by default, so the observability test fails on stable too.
This PR changes tests/, so CI runs all e2e jobs, and that job blocked the merge. So we can merge this asap.
Summary
Operators whose Neo4j or PostgreSQL database runs outside the Infrahub namespace can back up and restore that database with the
infrahub-backupchart. infrahub-backup 2.6.0 added this to the tool (opsmill/infrahub-backup#172). This PR sets the chart image to 2.6.0 and adds chart values for the settings that apply when the chart runs the tool.Key Changes
appVersionchanges from 2.4.1 to 2.6.0. See the 2.5.0 and 2.6.0 release notes.externalDatabasevalues. Each one sets the matching infrahub-backup environment variable:allowRestoresetsINFRAHUB_ALLOW_EXTERNAL_RESTORE, on the restore containers only.insecureTLSsetsINFRAHUB_EXTERNAL_DB_INSECURE_TLS.timeoutsetsINFRAHUB_EXTERNAL_DB_TIMEOUT.neo4jAddresssetsINFRAHUB_NEO4J_ADDRESS. For a Neo4j cluster, list every member.neo4jBackupPortsetsINFRAHUB_NEO4J_BACKUP_PORT.postgresAddresssetsINFRAHUB_POSTGRES_ADDRESS.restore.storage.s3.keynow also setsINFRAHUB_S3_BUCKET,INFRAHUB_S3_PREFIXandINFRAHUB_S3_ENDPOINT. A restore of an external Neo4j database copies the archive into this bucket, underseed/, and the Neo4j servers download it from there. So the credentials inrestore.storage.s3.secretNamemust be able to write to that bucket, and the Neo4j servers must be able to read from it. The download of the named key reads the bucket from itss3://address, so it does not change.restore.options.neo4jClusterOnlineTimeout. It passes--neo4j-cluster-online-timeout, which infrahub-backup 2.4.1 added.trace.sample_ratio), so the test often did not find the trace of its probe request in Tempo. Theobservability_k8sfixture now setsINFRAHUB_TRACE_SAMPLE_RATIO=1on the Infrahub server. The test fails onstablefor the same reason. It is in this PR because a change totests/runs every e2e job, and this job blocked the merge. It changes only the test, not a chart.What the chart does not have
neo4j-admin,cypher-shelland the PostgreSQL client in its own container and creates no Job and no Secret. The current Role covers this: the tool scales workloads withkubectl scale, which thedeployments/scaleandstatefulsets/scalerules cover.INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE, because the server does not report its store size. Set it withextraEnv. A larger volume for the temporary files can be added withextraVolumesandextraVolumeMounts, withTMPDIRset to its mount path inextraEnv.--external-db-image,--external-db-image-pull-secrets,--external-db-service-account,--external-db-node-selector,--external-db-tolerationsand--external-db-priority-class. They configure the temporary Job that the tool creates when it runs outside the Infrahub namespace, so they have no effect here.Related Context
docs/docs/backup/external-databases.mdxin infrahub-backup, section "Configure theinfrahub-backupHelm chart", describes how to use these values, with a backup and a restore example.Documentation Updates
charts/infrahub-backup/README.md, with the rows that helm-docs generates from thevalues.yamlcomments.tests/README.md: the new test in thebackuprow.charts/infrahub-backup/changelog/.Test Plan
make lintpasses on every chart.helm templaterenders the new variables in the Job and the CronJob of backup and restore. Backup containers do not getINFRAHUB_ALLOW_EXTERNAL_RESTORE.tests/e2e/test_infrahub_backup_external_neo4j.py, with thebackupmarker, so thee2e-infrahub-backupCI job runs it:neo4j.enabled=falsepointed at it.appVersion, with no extra arguments.INFRAHUB_EXTERNAL_DB_SCRATCH_SIZEis set withextraEnv.externalDatabase.allowRestore, must bring back the data from before the backup.uv run pytest -v -m backup tests/e2epassed locally: 3 tests in 10 minutes 53 seconds, the 2 tests oftest_infrahub_backup.pyand the new test.uv run pytest -v -m observability tests/e2epassed locally with the trace fix: 2 tests in 2 minutes 39 seconds.e2e-infrahub-backupwith the new test.neo4jAddress,neo4jBackupPort,postgresAddressandneo4jClusterOnlineTimeoutvalues.🤖 Generated with Claude Code