Problem
Since WAL-E was removed, configure_spilo.py writes the generated wal-g bucket prefix into a credentials variable in the wal-g env dir. Any deployment whose object-storage credentials come from the cloud metadata service rather than a key file loses WAL archiving and basebackups the moment it moves to an affected image.
Observed on ghcr.io/zalando/spilo-18:4.1-p2 (GKE, Workload Identity, GCS). Every archive_command fails:
ERROR: Failed to configure multi-storage: configure primary storage: configure storage with prefix
"gs://<bucket>/spilo/<scope>/wal/14": create Google Cloud storage: create GCS client: dialing:
google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable:
open gs://<bucket>/spilo/<scope>/wal/14: no such file or directory
because the env dir contains the prefix in both variables:
/run/etc/wal-e.d/env/WALG_GS_PREFIX = gs://<bucket>/spilo/<scope>/wal/14
/run/etc/wal-e.d/env/GOOGLE_APPLICATION_CREDENTIALS = gs://<bucket>/spilo/<scope>/wal/14
pg_stat_archiver shows archived_count = 0 with a climbing failed_count on every cluster. Pods stay healthy and Patroni reports the cluster as running, so nothing surfaces this except the archiver counters — the failure is silent until someone needs a restore.
There is no workaround possible via a configuration change.
Cause
prefix_env_name is write_envdir_names[0], and the compatibility block copies it to write_envdir_names[1]:
prefix_env_name = write_envdir_names[0]
...
# Set WALG_*_PREFIX for future compatibility
if store_type in ('S3', 'GS') and not walg.get(write_envdir_names[1]):
walg[write_envdir_names[1]] = walg[prefix_env_name]Before WAL-E was removed the lists began with the WAL-E prefix, so index 1 was the wal-g prefix. They now begin with the wal-g prefix:
gs_names = ['WALG_GS_PREFIX', 'GOOGLE_APPLICATION_CREDENTIALS']
s3_names = ['WALG_S3_PREFIX', 'AWS_ACCESS_KEY_ID', ...]
so index 1 is a credential. The copy is also redundant now that index 0 is already the wal-g prefix.
This is not GCS-specific — on S3 the same index is AWS_ACCESS_KEY_ID, so instance-role setups should be affected the same way.
Problem
Since WAL-E was removed,
configure_spilo.pywrites the generated wal-g bucket prefix into a credentials variable in the wal-g env dir. Any deployment whose object-storage credentials come from the cloud metadata service rather than a key file loses WAL archiving and basebackups the moment it moves to an affected image.Observed on
ghcr.io/zalando/spilo-18:4.1-p2(GKE, Workload Identity, GCS). Everyarchive_commandfails:because the env dir contains the prefix in both variables:
pg_stat_archivershowsarchived_count = 0with a climbingfailed_counton every cluster. Pods stay healthy and Patroni reports the cluster as running, so nothing surfaces this except the archiver counters — the failure is silent until someone needs a restore.There is no workaround possible via a configuration change.
Cause
prefix_env_nameiswrite_envdir_names[0], and the compatibility block copies it towrite_envdir_names[1]:Before WAL-E was removed the lists began with the WAL-E prefix, so index
1was the wal-g prefix. They now begin with the wal-g prefix:so index
1is a credential. The copy is also redundant now that index0is already the wal-g prefix.This is not GCS-specific — on S3 the same index is
AWS_ACCESS_KEY_ID, so instance-role setups should be affected the same way.