Skip to content

feat(infrahub-backup): support databases outside the Infrahub namespace - #125

Merged
danipl merged 11 commits into
stablefrom
dp-20261106-backup-external-databases-ifc-3553
Nov 16, 2026
Merged

danipl merged 11 commits into
stablefrom
dp-20261106-backup-external-databases-ifc-3553

Conversation

@danipl

@danipl danipl commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Operators whose Neo4j or PostgreSQL database runs outside the Infrahub namespace can back up and restore that database with the infrahub-backup chart. infrahub-backup 2.6.0 added this to the tool (opsmill/infrahub-backup#172). This PR sets the chart image to 2.6.0 and adds chart values for the settings that apply when the chart runs the tool.

Key Changes

  • Image 2.6.0. appVersion changes from 2.4.1 to 2.6.0. See the 2.5.0 and 2.6.0 release notes.
  • New externalDatabase values. Each one sets the matching infrahub-backup environment variable:
    • allowRestore sets INFRAHUB_ALLOW_EXTERNAL_RESTORE, on the restore containers only.
    • insecureTLS sets INFRAHUB_EXTERNAL_DB_INSECURE_TLS.
    • timeout sets INFRAHUB_EXTERNAL_DB_TIMEOUT.
    • neo4jAddress sets INFRAHUB_NEO4J_ADDRESS. For a Neo4j cluster, list every member.
    • neo4jBackupPort sets INFRAHUB_NEO4J_BACKUP_PORT.
    • postgresAddress sets INFRAHUB_POSTGRES_ADDRESS.
  • Staging bucket for a restore by key. A restore with restore.storage.s3.key now also sets INFRAHUB_S3_BUCKET, INFRAHUB_S3_PREFIX and INFRAHUB_S3_ENDPOINT. A restore of an external Neo4j database copies the archive into this bucket, under seed/, and the Neo4j servers download it from there. So the credentials in restore.storage.s3.secretName must be able to write to that bucket, and the Neo4j servers must be able to read from it. The download of the named key reads the bucket from its s3:// address, so it does not change.
  • New restore.options.neo4jClusterOnlineTimeout. It passes --neo4j-cluster-online-timeout, which infrahub-backup 2.4.1 added.
  • All new values are empty by default. Apart from the image tag and the S3 variables of a restore by key, the rendered chart does not change.
  • Fix of the observability e2e test. Infrahub 1.12 records 10% of traces by default (trace.sample_ratio), so the test often did not find the trace of its probe request in Tempo. The observability_k8s fixture now sets INFRAHUB_TRACE_SAMPLE_RATIO=1 on the Infrahub server. The test fails on stable for the same reason. It is in this PR because a change to tests/ runs every e2e job, and this job blocked the merge. It changes only the test, not a chart.

What the chart does not have

  • No new permissions. The backup and restore pods run in the Infrahub namespace. In that case, infrahub-backup runs neo4j-admin, cypher-shell and the PostgreSQL client in its own container and creates no Job and no Secret. The current Role covers this: the tool scales workloads with kubectl scale, which the deployments/scale and statefulsets/scale rules cover.
  • No value for the scratch size or a scratch volume. A backup of a Neo4j 2025 server needs INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE, because the server does not report its store size. Set it with extraEnv. A larger volume for the temporary files can be added with extraVolumes and extraVolumeMounts, with TMPDIR set to its mount path in extraEnv.
  • No values for --external-db-image, --external-db-image-pull-secrets, --external-db-service-account, --external-db-node-selector, --external-db-tolerations and --external-db-priority-class. They configure the temporary Job that the tool creates when it runs outside the Infrahub namespace, so they have no effect here.

Related Context

Documentation Updates

  • charts/infrahub-backup/README.md, with the rows that helm-docs generates from the values.yaml comments.
  • tests/README.md: the new test in the backup row.
  • Changelog fragments in charts/infrahub-backup/changelog/.

Test Plan

  • make lint passes on every chart.
  • helm template renders the new variables in the Job and the CronJob of backup and restore. Backup containers do not get INFRAHUB_ALLOW_EXTERNAL_RESTORE.
  • New e2e test tests/e2e/test_infrahub_backup_external_neo4j.py, with the backup marker, so the e2e-infrahub-backup CI job runs it:
    • It installs a Neo4j 2025.10.1 Enterprise server in its own namespace, and Infrahub with neo4j.enabled=false pointed at it.
    • The Jobs run the image of the chart's appVersion, with no extra arguments. INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE is set with extraEnv.
    • The backup Job must run the tools in its own container and create no Job, Pod or Secret.
    • The restore Job, with externalDatabase.allowRestore, must bring back the data from before the backup.
  • uv run pytest -v -m backup tests/e2e passed locally: 3 tests in 10 minutes 53 seconds, the 2 tests of test_infrahub_backup.py and the new test.
  • uv run pytest -v -m observability tests/e2e passed locally with the trace fix: 2 tests in 2 minutes 39 seconds.
  • CI passed on every e2e job, including e2e-infrahub-backup with the new test.
  • Not tested end to end: a Neo4j cluster, an external PostgreSQL database, and the neo4jAddress, neo4jBackupPort, postgresAddress and neo4jClusterOnlineTimeout values.

🤖 Generated with Claude Code

Daniel Pardo and others added 2 commits November 6, 2026 17:13
…ce [IFC-3553]

Add externalDatabase values for a Neo4j or PostgreSQL database that runs
outside the Infrahub namespace. Each one sets the matching infrahub-backup
environment variable: allowRestore (restore containers only), insecureTLS,
timeout, scratchSize, neo4jAddress, neo4jBackupPort and postgresAddress.
scratchVolume mounts a volume at /scratch and sets TMPDIR to it.

A restore by restore.storage.s3.key now also sets INFRAHUB_S3_BUCKET,
INFRAHUB_S3_PREFIX and INFRAHUB_S3_ENDPOINT. A restore of an external Neo4j
database copies the archive there for the Neo4j servers to download.

Add restore.options.neo4jClusterOnlineTimeout, which passes
--neo4j-cluster-online-timeout (infrahub-backup 2.4.1 or later).

All new values are empty by default, so the rendered chart does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the chart [IFC-3553]

Add a manual e2e test that installs a Neo4j 2025.10.1 Enterprise server in
its own namespace and Infrahub with neo4j.enabled=false pointed at it. The
backup and restore Jobs of the chart must run the database tools in their own
container, create no Job and no Secret, write their temporary files to an
ephemeral PersistentVolumeClaim, and bring back the data from before the
backup.

The test needs an infrahub-backup image built from
opsmill/infrahub-backup#172, so it is marked manual until that change is
released. Move the Garage setup into deploy_garage so both backup tests use
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bucket access [IFC-3553]

Bump appVersion to 2.4.1. restore.options.neo4jClusterOnlineTimeout passes
--neo4j-cluster-online-timeout, which infrahub-backup 2.4.0 does not have,
so a restore with this value failed with the default image.

Document that a restore of an external Neo4j database writes the staged
artifact to restore.storage.s3.bucket, also with restore.storage.s3.key. The
credentials must be able to write to that bucket, and the Neo4j servers must
be able to read from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tests/e2e/conftest.py Outdated
repository, _, tag = image.rpartition(":")

subprocess.run(["kubectl", "--kubeconfig", kubeconfig, "create", "namespace", namespace], check=True)
s3 = await deploy_garage(kubeconfig, namespace)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why aren't we using it as a fixture here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No good reason. Moved in 5c7d5ca: the new external_db_s3_k8s fixture creates the namespace and deploys Garage, as s3_k8s does for the community deployment, and infrahub_external_neo4j_k8s depends on it.

s3 = deployment["s3"]
chart = CHARTS_DIR / "infrahub-backup"
common = _common_sets(deployment)
image_arg = f"--external-db-image={deployment['backup_image']}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to test without --external-db-image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point. The Jobs run in direct mode, and infrahub-backup checks the image only when it creates a Job, so the flag was not needed. Removed in 218c007: the test now runs the chart with no extra arguments, as an operator does. The manual e2e test passes without the flag (1 passed in 5m47s).

volumeMounts:
- name: backup-storage
mountPath: {{ .Values.backup.storage.path }}
{{- include "infrahub-backup.scratchVolumeMount" . | nindent 16 }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the scratch volume is required for this PR, we can add it later if really required

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed in cc119cb. An operator who needs a larger volume for the temporary files can add it with extraVolumes and extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now says this. The manual e2e test no longer mounts a scratch volume, and it still passes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove all mentions of "scratch" in this PR

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5993a8a: the chart no longer has externalDatabase.scratchSize, and the chart has no other mention of "scratch". An operator who needs the size sets INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025 server needs it: the server does not report its store size, and infrahub-backup stops with an error when it has neither. The manual e2e test now sets the variable with extraEnv, so "scratch" is only left there, as the name of the variable. The test passes (1 passed in 6m09s).

Daniel Pardo and others added 3 commits November 9, 2026 09:56
…-3553]

The chart does not need its own value for the volume of the temporary files.
An operator who needs a larger volume can add it with extraVolumes and
extraVolumeMounts, and set TMPDIR with extraEnv. The scratchSize comment now
says so. The manual e2e test no longer mounts a scratch volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-db-image [IFC-3553]

The backup and restore Jobs run in the Infrahub namespace, so infrahub-backup
runs the database tools in its own container and creates no Job. It checks
the image of that Job only when it creates one, so the flag is not needed.
The test now runs the chart as an operator does, with no extra arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xture [IFC-3553]

Move the namespace and the Garage of the external Neo4j test into the
external_db_s3_k8s fixture, as s3_k8s does for the community deployment.
infrahub_external_neo4j_k8s now depends on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danipl
danipl marked this pull request as ready for review November 9, 2026 09:09

@fatih-acar fatih-acar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add some docs on how to use this feature using helm, in the infrahub-backup repo

Comment thread tests/e2e/conftest.py Outdated
EXTERNAL_NEO4J_AUTH_SECRET = "neo4j-auth"


def build_infrahub_backup_image(workdir: Path) -> str:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this once infrahub-backup is released with this new feature

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The follow-up PR that bumps appVersion to the infrahub-backup release with this feature removes the image build and moves the test to the backup marker, so CI runs it. No change in this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: infrahub-backup 2.6.0 is released, so this PR does it instead of a follow-up. 9fc97d5 removes the image build and moves the test to the backup marker. CI ran it in e2e-infrahub-backup and it passed.

…553]

The chart no longer has a value for the free disk space of a capture of an
external database. An operator who needs it sets
INFRAHUB_EXTERNAL_DB_SCRATCH_SIZE with extraEnv. A backup of a Neo4j 2025
server needs it, because the server does not report its store size.

The manual e2e test now sets the variable with extraEnv, in a values file
that also keeps the S3 CA settings: a --set on extraEnv would replace the
list of the CA values file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
danipl pushed a commit to opsmill/infrahub-backup that referenced this pull request Nov 9, 2026
Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how
to back up and restore an external database with the infrahub-backup
Helm chart.

Add a section to the external database guide that lists the chart's
externalDatabase values and the environment variable each one sets,
the settings the chart has no value for (scratch size, a larger
temporary volume), the restore requirements on the S3 bucket, and one
values example for a backup and one for a restore. Link it from the
prerequisites of the Kubernetes backup and restore guides, and point
the cluster online timeout troubleshooting at the new chart value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@@ -0,0 +1 @@
Bumped the `infrahub-backup` image to `2.4.1`, which `restore.options.neo4jClusterOnlineTimeout` needs. See the [infrahub-backup 2.4.1 release notes](https://github.com/opsmill/infrahub-backup/releases/tag/v2.4.1).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this changelog after having released a newer infrahub-backup

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a70bd88: the image is now 2.6.0, and +image-2.6.0.changed.md replaces the 2.4.1 fragment.

It links the 2.5.0 and 2.6.0 release notes.

danipl pushed a commit to opsmill/infrahub-backup that referenced this pull request Nov 16, 2026
* feat: back up and restore external databases on Kubernetes

Infrahub deployments on Kubernetes whose Neo4j Enterprise database,
PostgreSQL task-manager database, or both run outside the cluster had no
backup or recovery path through this toolset: every path executed the
vendor utility inside a database container, and there was none. This
supplies one — a short-lived Job in the namespace acting as the vendor's
own backup client — so the existing execution, streaming and copy
primitives work unchanged and a deployment whose databases are all
internal behaves exactly as before (FR-015).

Capture runs `neo4j-admin database backup --from=…` or `pg_dump` in that
workload and streams the artifact through the existing pipeline, so an
artifact taken externally is byte-compatible with an internal one and
restores by the same code. Where a database lives is settled at a gate
from positive evidence: a deployment query that fails is an error naming
the permission, never a silent switch to the external path, so a
permission problem cannot create workloads or pull images (FR-002).
Everything that would yield an unusable artifact is caught before data
moves — a utility older than the server, an external Community database,
missing workload permissions, a restore artifact the server cannot read.

Restore inverts the data path, because the server fetches the artifact
itself. The raw backup is staged in the configured object store, proven
readable by an independent client, seeded over Bolt, and then confirmed
online by polling: an accepted seed is not a completed one (FR-021). It
requires its own authorisation, distinct from --force and from enabling
scheduled restore (FR-009).

Nothing outlives the run. The workload carries its own deadline, its
credential Secret is garbage-collected with it, and a reaper removes
strays left by earlier runs (FR-011, FR-023). Connections are encrypted
and verified by default, with one operator opt-out (FR-029). Incomplete
captures are removed rather than retained, completeness is read from the
backup command's output rather than its exit status, and metadata
additions are additive so older releases keep restoring new artifacts
(FR-012, FR-016, FR-024).

Kubernetes resource ownership is decided in one place from label evidence
and declared release prefixes, replacing the substring matches that could
scale an unrelated workload to zero.

Spec, research, contracts and task ledger are archived under
specs/archive/007-external-database-backup/. The decisions are recorded
as ADRs 0008-0016, with operator documentation under docs/docs/backup/
and developer notes under dev/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(external): stop the run when workloads cannot be listed while locating a database

deploymentClaimsService reads Deployments and StatefulSets only once no
pod declares the service, which is exactly when they decide the answer: a
StatefulSet scaled to zero is internal only because its workload says so.
A failed workload listing was logged at debug level and skipped, so a
forbidden or unreachable listing returned "external" for a question that
was never answered. FR-002 names permission denied as a failure to query,
not as evidence.

listAllWorkloadsWith now returns the error, deploymentClaimsService and
locateServiceWith carry it into undeterminedLocation, and the RBAC hint
names deployments and statefulsets beside pods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): pull MinIO from pgsty/minio

minio/minio is no longer published on Docker Hub or quay.io, so every
S3 e2e test errored in fixture setup with ImageNotFound. pgsty/minio is
a community build of the same server; the fixture command and health
endpoint are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): bound external capture streams and fail a truncated one

A slow external stream that kept making progress never tripped the idle
timeout, so the transient pod's activeDeadlineSeconds could end it. The
pod's removal ends kubectl's stdout like a clean finish, and kloset
discards the reader's Close error, so the truncated archive was
committed as a complete snapshot.

execReadCloser now waits for the producing command on EOF and returns
its failure from Read, which kloset records as a source error and the
snapshot commit refuses. The external Neo4j and PostgreSQL streams are
also held to the capture's bound as a whole, the full-bound slot the
capture workload's deadline already budgets for the artifact leaving
the pod, so the tool ends a slow transfer before the cluster does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): gate a restore on the databases its archive carries

RestoreBackup settled where each database lives from --exclude-taskmanager
alone, before the archive was read. An archive without the task-manager
database was then refused, or given a transient workload, for an external
PostgreSQL the restore never writes. The gate now runs after the metadata
is parsed, on taskManagerIncluded && !excludeTaskManager, and still before
the edition probe, the wipe and the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(restore): gate the task-manager database on the archive's contents in every restore path

e200950 moved RestoreBackup's gate after the archive metadata is read. The
Plakar restore and the S3 leg of restore --latest still gated the
task-manager database on --exclude-taskmanager alone, before the backup was
read, so a backup without that database was refused, or given a transient
workload, for an external PostgreSQL the restore never writes.

The Plakar group restore now gates once the group is chosen, on whether it
holds a postgres snapshot; the single-snapshot restore gates once the
snapshot's component is read, before the edition probe. The S3 leg gates
only Neo4j before the download, which every archive carries, and leaves the
task manager to RestoreBackup's gate after the metadata is read; the second
pass over Neo4j prepares nothing again. All gates still run before anything
is stopped or wiped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): parse the keyword/value PostgreSQL connection form the way libpq does

statedKeywordFields split the keyword/value form on whitespace, so a quoted
value holding a space, such as sslrootcert='/etc/ssl/my certs/ca.crt', was
cut at the space, and whitespace around "=" lost the setting. Discovery then
recorded a truncated host or certificate path for the external task-manager
database.

The form is now parsed as libpq parses it: whitespace around "=", single-
quoted values, and backslash escapes for \' and \\. A string libpq would
refuse (an unterminated quote, a keyword without "=") is an error, and
discovery reads nothing from it rather than part of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): report each database's location even when the other cannot be located

environment detect resolved both databases through resolveDatabaseTargets,
which stops at the first location query that fails. One failed query hid the
other database from the report. The report now resolves each database on its
own through databaseTargetWith and logs a per-database warning; the run gate
still stops at the first unknown location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(neo4j): fail the cluster query in the subtest named for a failed query

The fake backend answered every cypher-shell call with success, so the
subtest exercised the parse of a successful answer and never the
assume-not-clustered branch it is named for. The fake can now fail
cypher-shell calls, and the subtest asserts the query was issued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(review): address review nitpicks

- Run golangci-lint v2.13.2 in CI, matching the dev-setup pin.
- Share the build tag between GO_TAGS and the lint target.
- Clear INFRAHUB_EXTERNAL_DB_INSECURE_TLS in the CLI test environment.
- Correct the spec directory path in plan.md, reconcile the open-task
  count in the implementation report, note where critique E6 was
  resolved, and align the backup artefact wording with FR-005.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plakar): keep a completed stream read past its EOF

c560786 made execReadCloser wait for the producing command when the
stream reaches EOF. Waiting closes the command's pipe, and kloset reads
once more after an EOF, so that read returned "file already closed".
kloset recorded it as a source error and every Plakar Neo4j snapshot was
refused, on Docker and Kubernetes alike.

Once the command has been waited for, Read now returns EOF. The new
test streams a real process through kloset, which the fake pipes in the
existing tests could not reproduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): address code-review findings on the external-database paths

- restore --latest --s3 resolves database credentials before gating, so
  the external Neo4j target carries the default database and user
- --reset-deployment-id runs through the restore pod when Neo4j is external
- a single Plakar postgres-snapshot restore restarts every service it
  stopped, and gates only the database it writes
- the restore pod's deadline covers every bound the pod is held through
- docker IsRunning treats a service the compose file lacks as not running
- the Prefect chart's component=server label no longer claims a pod for
  infrahub-server
- resolveJobPod falls back to polling when kubectl predates --for=create
- drop statedPrefectSSLRootCert, which only tests called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): count the deployment-ID reset in the restore workload deadline

When Neo4j is external the deployment-ID reset runs through the restore
workload, under the probe bound and retried up to ten times. That worst case
(about 50 minutes at the defaults) was not in the pod's activeDeadlineSeconds,
and the 10-minute margin cannot absorb it, so the cluster could remove the pod
during the reset and fail a restore that had already completed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): bound the job-pod poll listings by the remaining wait

The polling fallback for kubectl without --for=create ran each listing under
the control bound, independently of the poll's deadline. A listing started just
before the deadline could return after it, stretching pod resolution past its
ten-minute bound and accepting a pod found late. Each listing now runs through
queryWithin with the time left, and none is issued once the deadline passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): wait for the schema before handing a k8s test its URL

/api/config answers before infrahub-server can reach its database again
after the database pod reset between tests, so the next test's first SDK
call (the schema fetch) could get a 503. Wait for /api/schema as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(restore): wait for a healthy database before starting the app on docker

The Community streamed load resumes a Neo4j that then finishes its
shutdown and is restarted by its restart policy. `docker compose start
infrahub-server` checks the `service_healthy` dependency once and refuses
an exited database, so the post-restore start raced that restart and
failed with a bare "exit status 1".

All restore paths now start infrahub-server and task-worker through one
helper that, on Docker, waits (bounded, 5 min) for every database
container to be running and freshly healthy, starting it if it stays
exited. DockerBackend.Start and Stop include compose's output in their
errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(restore): bound each docker call of the database readiness wait

The five-minute wait only checked its deadline between readings, so a hung
docker compose ps, docker inspect or docker compose start held it
indefinitely. Each call now runs with the time left before the deadline and
a call cut short returns a timeout error naming it. runCommandContext also
bounds the wait for a killed command's output pipe, which a child such as the
compose plugin keeps open after the docker CLI is killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): wait for a database-backed query before handing a k8s test its URL

/api/schema can answer while infrahub-server still cannot reach Neo4j
after the database pod reset between tests, so the next test's first
mutation failed with 'Unable to connect to the database'. Wait for a
GraphQL query that reads the database instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(restore): kill the whole process group when a readiness docker call times out

A timeout killed only the docker CLI, so the compose plugin it runs as a
child could carry on with `docker compose start` after the wait had
reported it as timed out. The readiness wait's docker calls now run in a
process group of their own, and the timeout kills the group. Windows keeps
the single-process kill.

The fake docker scripts in the tests spawn `sleep 30` children; they now
die with the group instead of lingering for 30 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): retry Infrahub calls while it cannot reach its database

After the database pod is replaced, one successful read does not mean
every infrahub-server worker has reconnected: a later mutation still got
'Unable to connect to the database'. The readiness wait now needs three
reads in a row, and the seed, verify and modify helpers retry that
specific 503, which means the request did nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014fXAgGcEWm8mNkvxQVk538

* test(procgroup): count a zombie child as stopped

kill(pid, 0) still finds a killed child until whoever adopted it reaps
it, so the test could fail although the process-group kill worked. Ask
ps for the state and treat a gone or zombie process as not running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014fXAgGcEWm8mNkvxQVk538

* feat(external): reference the deployment's own Secret for database credentials

When the username and password a run holds were both read from the
deployment component's environment, and that component's running pod
declares the variables that held them as Secret keys (env secretKeyRef,
or a single matching envFrom secretRef), the transient Job now maps those
keys onto NEO4J_USERNAME/NEO4J_PASSWORD or PGUSER/PGPASSWORD with
secretKeyRef. No Secret is created, so no secrets verbs are needed.

The pod spec is read with a references-only jsonpath (pods get, already
granted), so an inline value never reaches the tool. PostgreSQL reuse
needs Secret-backed variables beside the Prefect URL that held the same
user and password, as prefect-server renders with
externalDatabase.usernameSecretKey/passwordSecretKey; a URL-only
deployment falls back.

Operator-supplied credentials, inline values, ConfigMaps, ambiguous
envFrom sources and unreadable pods fall back to the Job-owned Secret
unchanged. The stray reaper now warns instead of failing when listing
secrets is forbidden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(adr): state that only an external database needs the new permissions

ADR 0008 and 0009 listed the namespace grants first and the external-only
condition last, so they read as a requirement on every deployment. They now
lead with the condition, match the Job-based permissions the code needs, and
mark the secrets verbs as needed only when the deployment's own Secret keys
cannot be referenced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(adr): record the tools-in-image and Job-workload decisions

ADR 0017 moves the external-database tools into the infrahub-backup image,
as asked in review: the transient Job runs that image, and a tool already
running in the namespace runs them directly. ADR 0018 records the move from
a bare Pod to a Job, whose time-to-live lets the cluster remove the pod and
its credential Secret after a killed run. ADR 0008 and 0009 are marked as
partly superseded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): add the tools-in-image amendment (FR-030 to FR-033)

Requirements, plan section and task phase 6 for ADR 0017, asked for in
review of PR #172.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(image): ship neo4j-admin, cypher-shell and the PostgreSQL 18 client

The runtime image now carries the database tools the external-database
path needs (FR-030, ADR 0017):

- neo4j-admin and cypher-shell, with their jars and default conf, copied
  from neo4j:2025.10.1-enterprise into /opt/neo4j (NEO4J_HOME); NEO4J_CONF
  and NEO4J_LOGS point at directories the infrahub user owns
- Alpine's OpenJDK 21 (openjdk21-jre-headless=21.0.12_p8-r0)
- postgresql18-client=18.6-r0 (pg_dump, pg_restore, psql)

The base moves from alpine:3.21 to alpine:3.23: 3.21 and 3.22 package no
PostgreSQL 18 client. The Neo4j tools run on Alpine (musl), so the base
stays Alpine. The image still runs as infrahub (uid 1000).

scripts/check-image-tools.sh and `make docker-check-tools` build the image
and run each tool's --version as the image's user, failing when a tool is
missing or the image runs as root.

Local run, 2026-10-12T10:28Z, docker 29.4.0 linux/arm64:

  Image user: infrahub (uid 1000)
  ok   neo4j-admin: 2025.10.1
  ok   cypher-shell: Cypher-Shell 2025.10.1
  ok   pg_dump: pg_dump (PostgreSQL) 18.6
  ok   pg_restore: pg_restore (PostgreSQL) 18.6
  ok   psql: psql (PostgreSQL) 18.6
  Image size: 360619641 bytes (343 MiB)
  All database tools present in infrahub-backup:check-tools

Image size (docker image inspect .Size, linux/arm64):
  before: 34097838 bytes (33 MiB)
  after:  360619641 bytes (343 MiB)
linux/amd64 builds too (364076638 bytes) and its tools print the same
versions under emulation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(external): run the infrahub-backup image in the transient workload

The transient Job now runs this tool's own image, which carries
neo4j-admin, cypher-shell and the PostgreSQL 18 client (ADR 0017), in
place of the neo4j and postgres vendor images (T164-T168, FR-031, FR-033).

- One --external-db-image flag replaces --external-db-image-neo4j and
  --external-db-image-postgres. It stays unbound from the environment
  (FR-027). The old flags were never released, so they are removed.
- The default is registry.opsmill.io/opsmill/infrahub-backup:<version>,
  with the leading "v" of the release tag stripped, as the release
  workflow pushes it. A development build (empty version, commit hash,
  git describe output, build metadata) has no default and is refused in
  externalCaptureOps, before the stray reaper and before any service is
  stopped, by a message naming the flag.
- runAsUser, runAsGroup and fsGroup are 1000 for every role and both
  databases. The image declares its user by name, which the kubelet
  cannot check runAsNonRoot against, so the uid stays explicit.
- Docs, contract and knowledge notes describe one image to mirror.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(external): run the database tools in-process when infrahub-backup runs in the namespace

Direct mode (FR-032, ADR 0017, tasks T169 to T173). When the service-account
namespace file names the target namespace and every tool a database needs is
on PATH, externalCaptureOps returns operations served by a local workload: a
run-specific scratch directory under os.TempDir. No Job or Secret is created,
no run identifier is minted and the reaper does not run, so no jobs or
secrets permission is exercised. The decision follows the collect refusal and
precedes the image refusal, is taken once per database and is logged at info.

The capture and restore sequences do not fork. Their execution target becomes
local:<role>-<service>-<n>, and the seams they already call (execBounded,
ExecStreamPipe, copyAcrossTransientWorkload, and the new
refuseUnfittableRestoreDump) serve it in this process. The Job's /scratch mount
path is translated onto the scratch directory in argv, environment values and
copy paths, so the CA material lands there too. Credentials reach the child
only through its environment (FR-014). Scratch fit is checked against statfs
free space and the directory is removed on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): run the external suites on the image under test

The Kubernetes e2e job builds infrahub-backup:e2e-<sha> and exports
INFRAHUB_TESTING_EXTERNAL_DB_IMAGE. The external_db_image fixture loads
it into the docker-driver vcluster with ctr import, and every external
case passes it with --external-db-image, which is not bound to the
environment (FR-027).

The version-gate cases build an image from the one under test whose
neo4j-admin reports 4.0.0 or 2099.12.0, so both directions run against
the live server. The FR-023 case asserts the Secret is owned by the
Job and deletes the Job (ADR 0018); transient_objects lists Jobs.

A new case runs infrahub-backup as a Job in the namespace under the
direct-mode permissions and asserts a capture with no transient
workload (FR-032). Writing its Role showed the documented permissions
missed list on deployments/statefulsets and patch on their scale
resources; the docs are corrected.

The external suites still skip until T072 provisions the topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): assert the cluster-side cleanup guarantee after a killed run

A killed tool cannot delete its transient objects, and per ADR 0018 the
cluster removes them only after the Job's deadline and time-to-live. The
case now asserts what makes that happen: the Job carries a deadline and a
time-to-live, the pod belongs to the Job and restates the deadline, and any
credential Secret is owned by the Job. It then deletes the Job and waits
for the garbage collector to remove the pod and the Secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): tighten scratch translation, child environment and published-version check

- translate reads both boundaries of a /scratch occurrence in the original
  string, so /data/scratch/scratch is no longer rewritten.
- Direct-mode children no longer inherit PG*, NEO4J_* (other than the
  image's NEO4J_HOME, NEO4J_CONF, NEO4J_LOGS) or the JVM option variables
  from this process, matching what the Job's container receives.
- A git describe suffix after a pre-release (v2.4.0-rc.1-3-gabc) and a
  -dirty marker anywhere are refused as a published version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(image): trim the Java runtime and unused Neo4j jars

The image drops from 343.9 MiB to 208.7 MiB compressed (arm64). The Java
runtime is a jlink runtime with the modules the Neo4j tools use, and the
jars neo4j-admin and cypher-shell never load are removed: the browser, the
Cypher planner and runtimes, Scala, Parquet/Arrow, Jetty/Jersey/Prometheus/
JTS, and macOS/Windows native libraries.

Validated on arm64 against a real Neo4j 2025.10.1 Enterprise and
PostgreSQL 18: remote backup (plain and with a backup SSL policy), backup
inspect, restore, consistency check, cypher-shell over plain, self-signed
and verified TLS Bolt, pg_dump and pg_restore. amd64 could not run here, so
CI now runs scripts/check-image-tools.sh on the amd64 runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(image): drop the cloud storage client libraries from the Neo4j tools

The tool only ever passes neo4j-admin a local --to-path, and an S3 seed
restore is fetched by the Neo4j server itself, so the S3, Azure and Google
Cloud client jars are never loaded. Removing them takes the image from
208.7 MiB to 156.4 MiB compressed (arm64). Same validation as d1b3c2f; no
class is loaded from a removed jar. The direct-to-cloud procedure page now
says the image's neo4j-admin cannot write to cloud storage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): report the phase 6 implementation run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): stage each restore seed in a folder of its own

A Neo4j server seeding from a .backup URI reads the other artifacts beside it
as one backup chain and refuses the seed when they do not form one. Every
restore staged into one seed/ folder, and a failed restore keeps its staged
copy on purpose, so that copy made every later restore into the same bucket
fail with "not part of a valid backup chain". Observed on 2025.10.1 against
MinIO: the same artifact seeded alone and failed next to another.

The seed now goes to seed/<artifact name>/. Restoring the same archive twice
writes the same object again. The upload failure also states that nothing was
stopped, as the read-back failure already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kubernetes): report why the namespace check failed

With --k8s-namespace set, a refused or missing namespace was reported as
"exit status 1". The error now carries kubectl's message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): run the external suites against a real external topology

First run of the external suites (tasks.md T072, T175, T176). Corrections:
- INFRAHUB_TESTING_EXTERNAL_INSECURE_TLS and _SCRATCH_SIZE add the flags every
  external run needs: the tool verifies certificates by default and always
  encrypts PostgreSQL, and Neo4j 2025.x reports no store size (T158)
- the both-external overlay sets prefect-server.secret, without which the
  chart does not render
- the post-run check waits for the cluster to remove the Job's pod
- permission sets include workload listing and pod exec
- the undiscoverable-endpoint case waits for infrahub-server's pod to go
- source_roles is expected for Neo4j only; the component is task-manager-db

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the external-database suites against containers outside the cluster

The enterprise leg of e2e-tests-k8s starts Neo4j Enterprise, PostgreSQL and
MinIO on one Docker network, with a self-signed certificate on both servers,
and runs the external Neo4j and both-external sessions (tasks.md T072).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): record the first external end-to-end run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): stop waiting when the seeded database goes offline with a reason

Neo4j 2025.10.1 reports a rejected seed as offline with a statusMessage that
does not contain "Unable to start database", so the restore waited the whole
--external-db-timeout, two hours by default. An offline state with a message
now ends the wait; offline with no message still waits, because a database
being replaced can pass through it (tasks.md T178).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): one unreachable endpoint of several still captures the database

neo4j-admin 2025.10.1 exits 0 and writes a backup of the whole database when
one of two --from servers does not resolve, and FR-012 asks that every
requested database be captured. The case now asserts the archive is kept,
recorded as complete, and that the run warns about the endpoint it could not
reach (tasks.md T177).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): record the T177 and T178 decisions and the TLS issue

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): T072 passes in CI; record the external end-to-end run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(spec-007): bring task status and test comments in line with the CI run

The external suites now run in the enterprise leg of e2e-tests-k8s, so the
skip reason, module docstrings and task entries no longer say T072 is open.
T054 to T057 are ticked; T064 stays open because its Community case needs a
server this topology does not provide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): do not reference a Secret key reached only through envFrom

An envFrom source names no keys, and the kubelet skips a key the Secret
lacks, so a variable whose name matches a source's prefix (an empty
prefix matches every name) may come from the image instead. The
generated secretKeyRef is mandatory, so a missing key held the transient
pod pending and blocked the run. Checking the key would need get on
secrets, which the reference path exists to avoid, so such a variable
now falls back to the Job-owned Secret. Explicit env secretKeyRefs are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): record PostgreSQL provenance from the values the read applied

A partial operator connection URL (a user without a password) left the
credential check unsatisfied, so the task-manager read replaced both
values with the deployment's. The guard tested the values in place
before that read, so the deployment-sourced pair was never recorded and
the run fell back to creating a Job-owned Secret.

Record provenance when the deployment URL itself carried both the user
and the password, which applyPrefectConnection then applies. A URL
without a password still leaves the operator's in place and records
nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): make the direct-mode scratch path absolute

A relative TMPDIR gave a relative scratch path, which is both the child's
working directory and a path in its arguments, so once the child changed
into it every scratch argument resolved beneath itself. Resolve the parent
to an absolute path before creating the scratch directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): stop a direct-mode stream's child when the stream is closed

The child's context was cancelled only after cmd.Wait returned, so a stream
that hit its idle timeout or deadline closed stdout and then waited on a
child blocked in database I/O, which never notices the closed pipe. Closing
the stream now cancels the child first. Children run in their own process
group, so cancelling also stops what an `sh -c` pipeline started, and
WaitDelay bounds the wait for their output pipes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(external): bound direct-mode copies by --external-db-timeout

copyAcrossTransientWorkload returned before taking externalDBBound when the
target was a local workload, so a copy into or out of the scratch directory
on a slow or stalled filesystem could hold a backup or restore past the
bound every other external operation runs under. Local copies now take a
context and stop between reads once it is done, and the seam runs them
under the same bound, returning the timeout at the bound even when a copy
is stuck inside one read or write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the Kubernetes end-to-end job when only Python changed

The external suites are Python, so a change to them alone skipped the job that
runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(external): correct statements the review found inaccurate

- A missing Secret key is fixed by restoring the key; restarting the pods
  does not help.
- Direct mode also needs a temporary directory with a plain path.
- FR-031's refusal of a development build applies only to runs that create a
  transient workload.
- ADR 0008: the external path adds write permissions to those the Kubernetes
  path already needs. ADR 0018: the cleanup guarantee depends on the Job, TTL
  and garbage-collection controllers.
- The changelog says only named Secret keys avoid the secrets permission.
- The knowledge doc: a failed pod-spec read is not remembered, and the image to
  mirror is the one --external-db-image selects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: act only on the deployment --k8s-namespace or --project names

With --k8s-namespace set and the namespace check refused, detection fell
through to Docker. In CI this picked up another job's Compose project on the
runner, and the run started backing it up; a restore would have written to
it. An explicit target now pins its backend, and that backend's detection
failure is the run's failure (tasks.md T156).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(review): address the second wave of review comments

- e2e-tests-k8s needs go-lint directly, so a lint failure stops it.
- ADR 0008 names the scale resources as the only write kubectl scale makes.
- The configuration reference describes what happens when both --project and
  --k8s-namespace are set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(image): bump the OpenJDK pin to the version Alpine 3.23 ships

Alpine removed openjdk21-jdk and openjdk21-jmods 21.0.12_p8-r0 when it
published 21.0.13.0.3_p1-r0, so the image build stopped at the
java-runtime stage. Both e2e-tests-k8s jobs on the merge of main failed
there. scripts/check-image-tools.sh passes on the rebuilt image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): run backup and restore against Neo4j clusters

Every Kubernetes case ran against one Neo4j server, so the cluster
restore flow and the external path against a cluster had no end-to-end
coverage. Two topologies now run a three-member Neo4j cluster, built the
way Infrahub's HA example builds it: three Neo4j Helm releases and a
headless service, with Infrahub connecting through the neo4j protocol.

- internal: the cluster runs in the Infrahub namespace with the
  infrahub/service=database label. A tarball round trip checks that the
  restore takes the cluster flow and that the database is online on
  every member afterwards.
- external: the cluster runs in its own namespace, so the tool takes the
  external path. One case checks what the capture records for the
  cluster address, and one runs the full round trip, where the restore
  seeds every member from the object store.

The new e2e-tests-k8s-neo4j-cluster job runs each topology on its own
runner with the enterprise chart. Both topologies passed locally on a
docker-driver vcluster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): keep the release name with each Neo4j install process

mypy cannot index Popen.args, which python-lint reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): build the Neo4j cluster topologies in pytest on three nodes

Address the review of the Neo4j cluster tests:

- Create the vcluster with three nodes when a session declares a cluster
  topology, and keep the chart's anti-affinity, so each member runs on a
  node of its own. The install fails when the members share a node.
- Stage external restores in the session's Garage container (s3_server),
  attached to the vcluster's Docker network, instead of a MinIO container
  started by CI.
- Build the external topology in the fixtures: the address, the
  credentials, the flags and the infrahub-backup image. The CI job only
  declares the topology and runs pytest.

Add quality checks:

- After a restore, read every member on its own and check that all hold
  the same graph.
- Fail a session that declares a topology and passes none of its cases.
- Dump the cluster namespaces' logs when a cluster case fails.
- Upload the JUnit results of the cluster jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(e2e): back up and restore the Neo4j cluster from inside the namespace

Add a case to the external cluster suite that runs infrahub-backup in a
pod in the Infrahub namespace, the way the Helm chart runs it. The run
gets no --external-db-image, uses the tools of its own image (direct
mode, ADR 0017) and creates no transient Job. Its service account holds
only the direct-mode permissions. The case checks the backup and the
restore logs, that every member is online with the same graph, that the
pre-backup data is back, and that no transient object is left.

Move the direct-mode log lines into conftest so both suites share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(backup): configure the Helm chart for an external database

Fatih asked, in his review of opsmill/infrahub-helm#125, for docs on how
to back up and restore an external database with the infrahub-backup
Helm chart.

Add a section to the external database guide that lists the chart's
externalDatabase values and the environment variable each one sets,
the settings the chart has no value for (scratch size, a larger
temporary volume), the restore requirements on the S3 bucket, and one
values example for a backup and one for a restore. Link it from the
prerequisites of the Kubernetes backup and restore guides, and point
the cluster online timeout troubleshooting at the new chart value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(image): bump the PostgreSQL client pin to the version Alpine 3.23 ships

Alpine removed postgresql18-client 18.6-r0 when it published 18.7-r0, so
the image build stopped at the apk add step. e2e-tests-k8s (enterprise)
on the merge of main failed there. scripts/check-image-tools.sh passes
on the rebuilt image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Pardo <daniel@opsmill.com>
Daniel Pardo and others added 4 commits November 16, 2026 12:09
…external-databases-ifc-3553

# Conflicts:
#	tests/README.md
Set appVersion to 2.6.0, the first infrahub-backup release that backs up and
restores a database outside the Infrahub namespace, which the
externalDatabase values need. Replace the 2.4.1 changelog fragment, which
infrahub-backup-1.6.0 already released, with one for 2.6.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eased image [IFC-3553]

infrahub-backup 2.6.0 backs up and restores a database outside the Infrahub
namespace, and the chart now uses it. Remove the build of an image from
opsmill/infrahub-backup#172: the Jobs run the image of the chart's
appVersion. Move the test from the manual marker to the backup marker, so
the e2e-infrahub-backup CI job runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… e2e test

Infrahub 1.12 adds trace.sample_ratio, which records 10% of new traces by
default. The trace of the probe request of
test_grafana_can_query_infrahub_metrics_and_traces was then usually not
recorded, so Tempo had no trace to find and the test timed out. Set
INFRAHUB_TRACE_SAMPLE_RATIO=1 on the Infrahub server in the fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tests/e2e/conftest.py
Comment on lines +605 to +607
# Infrahub 1.12 records 10% of new traces by default; record all of
# them, so the trace of the probe request reaches Tempo.
"infrahubServer.infrahubServer.env.INFRAHUB_TRACE_SAMPLE_RATIO": "1",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it related to the current PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not directly. Infrahub 1.12 records only 10% of traces by default, so the observability test fails on stable too.

This PR changes tests/, so CI runs all e2e jobs, and that job blocked the merge. So we can merge this asap.

@danipl
danipl merged commit e1ca334 into stable Nov 16, 2026
13 checks passed
@danipl
danipl deleted the dp-20261106-backup-external-databases-ifc-3553 branch November 16, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants