Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
8df6a24
feat: back up and restore external databases on Kubernetes
BeArchiTek Sep 10, 2026
c2bcaeb
Merge remote-tracking branch 'origin/main' into 007-external-database…
BeArchiTek Oct 5, 2026
03ba244
fix(external): stop the run when workloads cannot be listed while loc…
BeArchiTek Oct 5, 2026
fb213e2
test(e2e): pull MinIO from pgsty/minio
BeArchiTek Oct 5, 2026
c560786
fix(external): bound external capture streams and fail a truncated one
BeArchiTek Oct 5, 2026
e200950
fix(external): gate a restore on the databases its archive carries
BeArchiTek Oct 5, 2026
e0cab74
fix(restore): gate the task-manager database on the archive's content…
BeArchiTek Oct 5, 2026
3881bb0
fix(external): parse the keyword/value PostgreSQL connection form the…
BeArchiTek Oct 5, 2026
2242579
fix(environment): report each database's location even when the other…
BeArchiTek Oct 5, 2026
23be87e
test(neo4j): fail the cluster query in the subtest named for a failed…
BeArchiTek Oct 5, 2026
3584a53
chore(review): address review nitpicks
BeArchiTek Oct 5, 2026
f24c0ed
fix(plakar): keep a completed stream read past its EOF
BeArchiTek Oct 5, 2026
0a6eb71
fix(external): address code-review findings on the external-database …
BeArchiTek Oct 5, 2026
d632fff
fix(external): count the deployment-ID reset in the restore workload …
BeArchiTek Oct 5, 2026
79f278b
fix(external): bound the job-pod poll listings by the remaining wait
BeArchiTek Oct 5, 2026
da1b884
test(e2e): wait for the schema before handing a k8s test its URL
BeArchiTek Oct 5, 2026
25941e9
fix(restore): wait for a healthy database before starting the app on …
BeArchiTek Oct 5, 2026
bc6321b
fix(restore): bound each docker call of the database readiness wait
BeArchiTek Oct 5, 2026
97b2af0
test(e2e): wait for a database-backed query before handing a k8s test…
BeArchiTek Oct 5, 2026
d4bb6b5
fix(restore): kill the whole process group when a readiness docker ca…
BeArchiTek Oct 5, 2026
72eb2d3
test(e2e): retry Infrahub calls while it cannot reach its database
BeArchiTek Oct 5, 2026
241faa4
test(procgroup): count a zombie child as stopped
BeArchiTek Oct 5, 2026
8981548
feat(external): reference the deployment's own Secret for database cr…
BeArchiTek Oct 12, 2026
887fa50
docs(adr): state that only an external database needs the new permiss…
BeArchiTek Oct 12, 2026
263ad9a
docs(adr): record the tools-in-image and Job-workload decisions
BeArchiTek Oct 12, 2026
a170ef0
docs(spec-007): add the tools-in-image amendment (FR-030 to FR-033)
BeArchiTek Oct 12, 2026
ea1d73b
feat(image): ship neo4j-admin, cypher-shell and the PostgreSQL 18 client
BeArchiTek Oct 12, 2026
48f6e95
feat(external): run the infrahub-backup image in the transient workload
BeArchiTek Oct 12, 2026
b5e87ad
feat(external): run the database tools in-process when infrahub-backu…
BeArchiTek Oct 12, 2026
e6abd08
test(e2e): run the external suites on the image under test
BeArchiTek Oct 12, 2026
90a088c
test(e2e): assert the cluster-side cleanup guarantee after a killed run
BeArchiTek Oct 13, 2026
97461b9
fix(external): tighten scratch translation, child environment and pub…
BeArchiTek Oct 13, 2026
d1b3c2f
build(image): trim the Java runtime and unused Neo4j jars
BeArchiTek Oct 13, 2026
3fe7b95
build(image): drop the cloud storage client libraries from the Neo4j …
BeArchiTek Oct 13, 2026
2a506b0
docs(spec-007): report the phase 6 implementation run
BeArchiTek Oct 13, 2026
9ba780d
fix(external): stage each restore seed in a folder of its own
BeArchiTek Oct 13, 2026
16c4a08
fix(kubernetes): report why the namespace check failed
BeArchiTek Oct 13, 2026
afa67f1
test(e2e): run the external suites against a real external topology
BeArchiTek Oct 13, 2026
1bb78fe
ci: run the external-database suites against containers outside the c…
BeArchiTek Oct 13, 2026
2ebc56e
docs(spec-007): record the first external end-to-end run
BeArchiTek Oct 13, 2026
5707324
fix(external): stop waiting when the seeded database goes offline wit…
BeArchiTek Oct 13, 2026
ca4992e
test(e2e): one unreachable endpoint of several still captures the dat…
BeArchiTek Oct 13, 2026
2bfed31
docs(spec-007): record the T177 and T178 decisions and the TLS issue
BeArchiTek Oct 13, 2026
a5a2ee5
docs(spec-007): T072 passes in CI; record the external end-to-end run
BeArchiTek Oct 13, 2026
d30545f
docs(spec-007): bring task status and test comments in line with the …
BeArchiTek Oct 13, 2026
76d674f
fix(external): do not reference a Secret key reached only through env…
BeArchiTek Oct 13, 2026
395e062
fix(external): record PostgreSQL provenance from the values the read …
BeArchiTek Oct 13, 2026
43eace0
fix(external): make the direct-mode scratch path absolute
BeArchiTek Oct 13, 2026
9bd34bc
fix(external): stop a direct-mode stream's child when the stream is c…
BeArchiTek Oct 13, 2026
4737a16
fix(external): bound direct-mode copies by --external-db-timeout
BeArchiTek Oct 13, 2026
3f9abb2
ci: run the Kubernetes end-to-end job when only Python changed
BeArchiTek Oct 13, 2026
ad45562
docs(external): correct statements the review found inaccurate
BeArchiTek Oct 13, 2026
7090010
fix: act only on the deployment --k8s-namespace or --project names
BeArchiTek Oct 13, 2026
304756b
chore(review): address the second wave of review comments
BeArchiTek Oct 13, 2026
c11c057
Merge main into 007-external-database-backup
Nov 6, 2026
742add6
build(image): bump the OpenJDK pin to the version Alpine 3.23 ships
Nov 6, 2026
e9fd83b
test(e2e): run backup and restore against Neo4j clusters
Nov 6, 2026
8f96843
test(e2e): keep the release name with each Neo4j install process
Nov 6, 2026
01a308c
test(e2e): build the Neo4j cluster topologies in pytest on three nodes
Nov 6, 2026
91eafee
test(e2e): back up and restore the Neo4j cluster from inside the name…
Nov 6, 2026
69f368b
docs(backup): configure the Helm chart for an external database
Nov 9, 2026
b9e9b23
Merge PR #199: Neo4j cluster e2e tests
Nov 16, 2026
8af0bbd
Merge branch 'main' into 007-external-database-backup
Nov 16, 2026
9a61410
build(image): bump the PostgreSQL client pin to the version Alpine 3.…
Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 192 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.7.2
version: v2.13.2

test:
if: needs.files-changed.outputs.go == 'true'
Expand Down Expand Up @@ -130,12 +130,20 @@ jobs:
PYTEST_MARKERS: ${{ matrix.edition == 'enterprise' && 'docker' || 'docker and not enterprise' }}
run: exec tini -s -g -- uv run pytest -v -m "$PYTEST_MARKERS" tests/e2e

# Python changes run it too: the suites themselves are Python, and a change to them alone
# must still be exercised. `test` is skipped when no Go changed, and a skipped need skips
# this job unless the condition says otherwise, so it runs unless a need failed.
e2e-tests-k8s:
if: needs.files-changed.outputs.go == 'true'
needs: ["files-changed", "test"]
if: |
!cancelled() && !contains(needs.*.result, 'failure') &&
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
(needs.files-changed.outputs.go == 'true' || needs.files-changed.outputs.python == 'true')
# go-lint is listed so that its failure stops this job; `test` only inherits it as a skip.
needs: ["files-changed", "go-lint", "test"]
runs-on:
group: huge-runners
timeout-minutes: 30
# The enterprise leg also runs the two external-database sessions, each with its own
# vcluster and Infrahub deployment.
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -173,6 +181,91 @@ jobs:
- name: Build binaries (infrahub-backup, infrahub-taskmanager, infrahub-collect)
run: make build

# The external-database cases run the database tools from this image (ADR 0017) and
# pass it with --external-db-image, because a development build has no published image
# to default to. The vcluster is created by pytest, so the `external_db_image` fixture
# loads the image into the cluster's nodes; it is tagged with the commit rather than
# `latest` so that the workload takes it from the node instead of pulling it.
- name: Build the infrahub-backup image under test
run: |
docker build -t "infrahub-backup:e2e-${GITHUB_SHA}" .
echo "INFRAHUB_TESTING_EXTERNAL_DB_IMAGE=infrahub-backup:e2e-${GITHUB_SHA}" >> "$GITHUB_ENV"

# The image's Java runtime and Neo4j jars are trimmed to what the tools load. This runner is
# amd64, so this is where the trimmed image is checked on that architecture.
- name: Check the image carries working database tools
run: SKIP_BUILD=1 ./scripts/check-image-tools.sh "infrahub-backup:e2e-${GITHUB_SHA}"

# The external-database suites (tasks.md T072) need Neo4j Enterprise, PostgreSQL and an
# object store outside the cluster under test. They run as containers on one Docker
# network, so Neo4j can fetch a restore's seed from MinIO by name. Pods in the vcluster
# and the tool on the runner both reach them through the Docker host's bridge address,
# where the ports are published. Only the enterprise leg declares the topology, so the
# suites run once.
- name: Start the external databases
if: matrix.edition == 'enterprise'
run: |
docker rm -f ext-minio ext-neo4j ext-postgres 2>/dev/null || true
docker network rm ext-db 2>/dev/null || true
docker network create ext-db
# The tool verifies certificates by default (FR-029) and always encrypts PostgreSQL,
# so both servers present one self-signed certificate and the suites opt out of
# verifying it. Neo4j keeps plain Bolt available for Infrahub's own connections.
certs="${RUNNER_TEMP}/ext-db-certs"
rm -rf "${certs}" && mkdir -p "${certs}/bolt/trusted" "${certs}/bolt/revoked"
openssl req -x509 -newkey rsa:2048 -nodes -days 2 -subj "/CN=ext-db" \
-keyout "${certs}/bolt/private.key" -out "${certs}/bolt/public.crt"
cp "${certs}/bolt/public.crt" "${certs}/bolt/trusted/"
chmod -R a+rX "${certs}"
docker run -d --name ext-minio --network ext-db -p 19000:9000 \
-e MINIO_ROOT_USER=extaccess -e MINIO_ROOT_PASSWORD=extsecretkey \
--entrypoint sh pgsty/minio:RELEASE.2026-08-04T00-00-00Z \
-c 'mkdir -p /data/infrahub-seed && minio server /data'
docker run -d --name ext-neo4j --network ext-db -p 17687:7687 -p 6362:6362 \
-v "${certs}:/ssl:ro" \
-e NEO4J_dbms_ssl_policy_bolt_enabled=true \
-e NEO4J_dbms_ssl_policy_bolt_base__directory=/ssl/bolt \
-e NEO4J_dbms_ssl_policy_bolt_client__auth=NONE \
-e NEO4J_server_bolt_tls__level=OPTIONAL \
-e NEO4J_ACCEPT_LICENSE_AGREEMENT=yes -e NEO4J_AUTH=neo4j/admin12345 \
-e NEO4J_server_backup_enabled=true \
-e NEO4J_server_backup_listen__address=0.0.0.0:6362 \
-e AWS_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=extaccess -e AWS_SECRET_ACCESS_KEY=extsecretkey \
-e AWS_ENDPOINT_URL_S3=http://ext-minio:9000 \
neo4j:2025.10.1-enterprise
docker run -d --name ext-postgres --network ext-db -p 15432:5432 \
-v "${certs}:/certs:ro" \
-e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=prefect -e POSTGRES_DB=prefect \
--entrypoint sh postgres:18-alpine -c '
mkdir -p /etc/pgssl && cp /certs/bolt/public.crt /certs/bolt/private.key /etc/pgssl/ &&
chown -R postgres /etc/pgssl && chmod 600 /etc/pgssl/private.key &&
exec docker-entrypoint.sh postgres -c ssl=on -c ssl_cert_file=/etc/pgssl/public.crt -c ssl_key_file=/etc/pgssl/private.key'
for i in $(seq 1 90); do
docker exec ext-neo4j cypher-shell -u neo4j -p admin12345 'RETURN 1' && break
sleep 2
done
for i in $(seq 1 30); do
docker exec ext-postgres pg_isready -U postgres && break
sleep 1
done
host=$(docker network inspect bridge --format '{{(index .IPAM.Config 0).Gateway}}')
{
echo "INFRAHUB_TESTING_EXTERNAL_NEO4J_ADDRESS=${host}:17687"
echo "INFRAHUB_TESTING_EXTERNAL_NEO4J_USERNAME=neo4j"
echo "INFRAHUB_TESTING_EXTERNAL_NEO4J_PASSWORD=admin12345"
echo "INFRAHUB_TESTING_EXTERNAL_NEO4J_EDITION=enterprise"
echo "INFRAHUB_TESTING_EXTERNAL_POSTGRES_ADDRESS=${host}:15432"
echo "INFRAHUB_TESTING_EXTERNAL_POSTGRES_USERNAME=postgres"
echo "INFRAHUB_TESTING_EXTERNAL_POSTGRES_PASSWORD=prefect"
echo "INFRAHUB_TESTING_EXTERNAL_SEED_BUCKET=infrahub-seed"
echo "INFRAHUB_TESTING_EXTERNAL_SEED_ENDPOINT=http://${host}:19000"
echo "INFRAHUB_TESTING_EXTERNAL_SEED_ACCESS_KEY=extaccess"
echo "INFRAHUB_TESTING_EXTERNAL_SEED_SECRET_KEY=extsecretkey"
echo "INFRAHUB_TESTING_EXTERNAL_INSECURE_TLS=true"
# Neo4j 2025.x reports no store size, so a capture needs one (tasks.md T158).
echo "INFRAHUB_TESTING_EXTERNAL_SCRATCH_SIZE=2Gi"
} >> "$GITHUB_ENV"

- name: Install Python dependencies
run: uv sync --all-packages

Expand All @@ -184,6 +277,101 @@ jobs:
- name: Run e2e tests (Kubernetes)
run: exec tini -s -g -- uv run pytest -v -m k8s tests/e2e

# Each external topology is its own pytest session, because the deployment under test

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

The preceding all-tests/e2e pytest step also collects the new external suites, so on the enterprise leg they run once there and again in each dedicated session. Exclude these files from the generic run or remove the duplicate invocation; otherwise this comment's claim that each topology runs once is false.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/ci.yml, line 274:

<comment>The preceding all-`tests/e2e` pytest step also collects the new external suites, so on the enterprise leg they run once there and again in each dedicated session. Exclude these files from the generic run or remove the duplicate invocation; otherwise this comment's claim that each topology runs once is false.</comment>

<file context>
@@ -199,6 +271,26 @@ jobs:
       - name: Run e2e tests (Kubernetes)
         run: exec tini -s -g -- uv run pytest -v -m k8s tests/e2e
 
+      # Each external topology is its own pytest session, because the deployment under test
+      # is created once per session from INFRAHUB_TESTING_EXTERNAL_DB.
+      - name: Run e2e tests (Kubernetes, external Neo4j)
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this. The generic step does collect the external suites, but it does not set INFRAHUB_TESTING_EXTERNAL_DB, so every external case skips there. In the CI run of 2bfed31 (run) the generic step reported 5 passed, 17 skipped, and the external cases ran only in the two dedicated sessions.

# is created once per session from INFRAHUB_TESTING_EXTERNAL_DB.
- name: Run e2e tests (Kubernetes, external Neo4j)
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
if: matrix.edition == 'enterprise'
env:
INFRAHUB_TESTING_EXTERNAL_DB: database
run: exec tini -s -g -- uv run pytest -v -m k8s tests/e2e/test_k8s_external_neo4j.py tests/e2e/test_k8s_external_in_namespace.py

- name: Run e2e tests (Kubernetes, external Neo4j and PostgreSQL)
if: matrix.edition == 'enterprise'
env:
INFRAHUB_TESTING_EXTERNAL_DB: database,task-manager-db
run: exec tini -s -g -- uv run pytest -v -m k8s tests/e2e/test_k8s_external_both.py

- name: Remove the external databases
if: always() && matrix.edition == 'enterprise'
run: |
docker rm -f ext-minio ext-neo4j ext-postgres 2>/dev/null || true
docker network rm ext-db 2>/dev/null || true

# A three-member Neo4j cluster, built like Infrahub's HA example, once inside the
# Infrahub namespace and once outside it. Every other Kubernetes case runs against one
# Neo4j server. Each topology runs on its own runner, because one cluster and one
# Infrahub deployment already fill one.
e2e-tests-k8s-neo4j-cluster:
if: |
!cancelled() && !contains(needs.*.result, 'failure') &&
(needs.files-changed.outputs.go == 'true' || needs.files-changed.outputs.python == 'true')
needs: ["files-changed", "go-lint", "test"]
runs-on:
group: huge-runners
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
topology: [internal, external]
env:
INFRAHUB_HELM_CHART: oci://registry.opsmill.io/opsmill/chart/infrahub-enterprise
INFRAHUB_TESTING_NEO4J_CLUSTER: ${{ matrix.topology }}
steps:
- uses: actions/checkout@v6

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.25"

- name: Set up vCluster
uses: loft-sh/setup-vcluster@68e17a8c3618e3c47d635842b803d4f560f694e8

- name: Set up Helm
uses: azure/setup-helm@v4

- name: Set up kubectl
uses: azure/setup-kubectl@v4

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.12"

- name: Set up Python and uv
uses: astral-sh/setup-uv@v7

- name: Install tini
run: apt-get update && apt-get install -y tini

- name: Build binaries (infrahub-backup, infrahub-taskmanager, infrahub-collect)
run: make build

- name: Install Python dependencies
run: uv sync --all-packages

- name: "Set environment variables"
run: |
RUNNER_NAME=$(echo "${{ runner.name }}" | grep -o 'ghrunner[0-9]\+' | sed 's/ghrunner\([0-9]\+\)/ghrunner_\1/')
echo "VCLUSTER_DIR=/var/lib/github/${RUNNER_NAME}/_temp/.vcluster/config.json" >> $GITHUB_ENV

# The fixtures build the whole topology: the three-node vcluster, the Neo4j cluster,
# the object store and the infrahub-backup image (tests/e2e/conftest.py, "Neo4j
# cluster topologies"). The cases of the other topology skip.
- name: Run e2e tests (Kubernetes, Neo4j cluster)
run: >-
exec tini -s -g -- uv run pytest -v -rs -m k8s
tests/e2e/test_k8s_neo4j_cluster.py tests/e2e/test_k8s_external_neo4j_cluster.py
--junitxml=neo4j-cluster-${{ matrix.topology }}-results.xml

- name: Upload the test results
if: always()
uses: actions/upload-artifact@v4
with:
name: neo4j-cluster-${{ matrix.topology }}-results
path: neo4j-cluster-${{ matrix.topology }}-results.xml
if-no-files-found: ignore

nix-build:
if: needs.files-changed.outputs.nix == 'true'
needs: ["files-changed"]
Expand Down
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,11 @@ Thumbs.db

# Spec-kit local state (records the active feature per checkout; not shared)
.specify/feature.json

# Spec-kit extension caches
.specify/**/.cache/

# Plakar/kloset writes a version-named repository directory into the working
# directory when the e2e suite drives it (e.g. `2.0.0/`). It is test residue, not
# source: a `git add -A` after an e2e run otherwise captures ~30 of its files.
/[0-9]*.[0-9]*.[0-9]*/
Loading
Loading