Skip to content

feat(collect): report task manager database storage in bundles - #193

Open
fatih-acar wants to merge 28 commits into
mainfrom
fac/report-taskmgr-storage-usage-d8y7s
Open

fatih-acar wants to merge 28 commits into
mainfrom
fac/report-taskmgr-storage-usage-d8y7s

Conversation

@fatih-acar

@fatih-acar fatih-acar commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Why

A bundle from infrahub-collect create does not show how much disk space the task manager's PostgreSQL database (task-manager-db) uses, or which records use it. Support then has to send the customer SQL queries to run (#181).

With this change, support can answer these questions from the bundle alone (PRD, spec):

  1. How big is the database?
  2. Which tables use the space?
  3. Is that space live rows, or deleted rows that VACUUM has not reclaimed yet?
  4. Within the tables that hold flow runs, task runs, states, logs, events and artifacts, which kinds of records use the space?

Every bundle answers questions 1 to 3. Question 4 needs the new --include-task-manager-storage flag, because answering it reads every row of those tables.

Out of scope, as listed in the spec:

  • External PostgreSQL servers, which will follow #172.
  • Byte sizes for each group of records in the breakdown.
  • Free space on the database volume, and growth over time.
  • Interpreting the figures, or running VACUUM or a flush.
  • Neo4j storage figures, and Prefect 2.x deployments.

Closes #181

What changed

Storage figures in every bundle

  • A new collector, task-manager-db-storage, writes task-manager-db/storage.json:
    • the database size, and when PostgreSQL last reset its statistics;
    • for each table, its data, index and TOAST sizes, its live and dead row estimates, and its last vacuum times.
  • The query has a 60-second limit.

Storage breakdown with --include-task-manager-storage

  • The flag can also be set with INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.
  • A second collector writes task-manager-db/storage-breakdown.json. It covers seven Prefect tables: events, flow_run, task_run, flow_run_state, task_run_state, log and artifact.
  • For each table, it gives the exact row count and the oldest and newest timestamps of each group of records. The groups are, for example, flow name and state type for flow runs, or flow name and log level for logs.
  • Each table's query has a 5-minute limit. A table that reaches the limit is recorded as failed, and the other tables are still collected. A table that the deployed Prefect version does not have is recorded as absent.

Failures

  • A failure never stops the bundle. The manifest records the reason, for example timed out after 60s, login rejected: ..., psql missing: ... or not reachable: ....
  • The diagnostics go to storage.err.txt or storage-breakdown.err.txt, with the password masked.
  • Both collectors are skipped when the deployment has no task-manager-db service.

Change to infrahub-backup

  • infrahub-backup reads the user, password and database only from the Prefect connection URL itself.
  • A part that the URL leaves out takes the default. Before this change, it took a value from PGUSER, PGPASSWORD, PGDATABASE, ~/.pgpass or the operating-system user of the machine that runs the tool.
  • The warning about a URL that does not parse now names the part that failed, without quoting the URL.

Implementation notes

  • Every query goes through one runner, runTaskManagerDBQuery, which runs psql inside task-manager-db (ADR 0008):
    • The session is read-only.
    • PostgreSQL enforces the time limit (statement_timeout), so a query stops on the server even after the tool stops waiting for it.
    • The breakdown queries turn off hash joins, merge joins and parallel workers, so they write no temporary files to a volume that may be nearly full.
  • The password reaches psql on stdin, through a short sh script, so it is in no argument of the docker, kubectl, sh or psql process.
  • The tool looks for the login in this order:
    1. a Prefect connection URL in the operator's environment;
    2. a Prefect connection URL in the task-manager container;
    3. POSTGRES_USER or POSTGRES_PASSWORD in task-manager-db;
    4. the defaults of the backup tool.
  • The Prefect connection URL is parsed with url.Parse instead of pgx.ParseConfig, so github.com/jackc/pgx/v5 is no longer a dependency. The Nix vendorHash is updated.
  • On a CloudNativePG cluster, the queries run in the primary pod.

What stayed the same

  • The backup archive format and infrahub-backup create are unchanged. infrahub-backup create still passes the database password to pg_dump as a docker or kubectl argument.
  • The other collectors and the existing infrahub-collect flags are unchanged.

Extraction of the spec into dev/

The last commit, 2380139, runs speckit-opsmill-extract on the spec, so the PR lands with the knowledge extracted. The PR had already written ADR 0008, the storage section of the knowledge page and the timeout-wrapping exception of the guidelines; the commit adds what was still missing:

  • ADR 0009: PostgreSQL is reached only through psql, pg_dump and pg_restore inside task-manager-db, over TCP to localhost, with the alternatives that research R1 rejected (a Go client, Unix-socket peer authentication, the Prefect server container).
  • dev/knowledge/infrahub-collect.md: the file contracts of storage.json and storage-breakdown.json, the manifest reason for each outcome, the five error kinds and the progress lines, the privileges the queries need, and a gotcha: the official postgres image trusts localhost, so a wrong password still logs in on Docker Compose.
  • dev/guidelines/collectors.md: one JSON value per query, the breakdown list as data, and how to test a database time limit and a change to the password path end to end.
  • The spec moved to specs/archive/006-taskmgr-db-storage/ with its extraction record. The references to it in ADR 0008, two Go comments and tests/helpers/bundle.py point at the archive.

Suggested review order

  1. ADR 0008 and the spec, then ADR 0009.
  2. src/internal/app/collect_taskmanager_db.go: credential lookup, query runner, error classification and masking.
  3. src/internal/app/collect_storage.go: the SQL and the two collectors.
  4. src/internal/app/app_config.go: the Prefect URL parsing, which infrahub-backup also uses.
  5. ExecSeparateInputContext in environment_docker.go and environment_kubernetes.go.
  6. The unit tests, then tests/e2e/test_docker_collect.py and tests/e2e/test_k8s_collect.py.

The files under specs/archive/006-taskmgr-db-storage/ are planning records. They can be skimmed. The last commit extracted the spec into dev/ and archived it (see "Extraction of the spec into dev/" above).

How to review

These parts need the most attention:

  • The sh script that reads the password from stdin. On Kubernetes it depends on kubectl exec -i. This path has unit tests only: no end-to-end run has covered it yet (see "How to test").
  • Locks. Each breakdown query holds an ACCESS SHARE lock on the table it reads and on the tables it joins, for up to 5 minutes. During a task manager upgrade, a schema change on one of these tables waits until the query ends. The user documentation warns about this.

These review findings are not addressed in this PR:

  • The collect tool has its own credential lookup, with a different fallback order from the backup tool's. When the Prefect server is down, infrahub-collect can log in to task-manager-db while infrahub-backup, and --include-backup, fall back to the defaults.
  • On Kubernetes, the pod name is looked up once, and every query of the breakdown reuses it. If CloudNativePG moves the primary to another pod during the breakdown, which can take up to 35 minutes, the remaining queries go to the old pod.
  • The tool uses the user, password and database of the Prefect URL, but not its host and port. It always connects to localhost:5432 inside task-manager-db.
  • The breakdown does not include Prefect 3's event_resources table.

The implementation report lists the other deferred findings.

How to test

make build-watchdog   # once per worktree
make test
make vet
GOTOOLCHAIN=go1.25.0 go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.7.2 run
uv run pytest -v -m docker tests/e2e/test_docker_collect.py
uv run pytest -v -m k8s tests/e2e/test_k8s_collect.py

Results:

  • make test, make vet and golangci-lint (0 issues) pass on 419cd9f, and so do rumdl, Vale and nix build.
  • The Docker and Kubernetes end-to-end tests passed on 6f21171, before the last two commits. The last two commits change how the password reaches psql, and CI runs both end-to-end suites on this PR.
  • The stdin path was checked through the real Docker backend against a postgres:18-alpine container that requires a password on localhost:
    • A password with spaces, quotes, @ and $ logs in.
    • The session is read-only and named infrahub-collect.
    • No host process shows the password in its arguments while a query runs.
    • A wrong password is reported as login rejected.
  • The CloudNativePG primary-pod check is recorded in t046-cnpg-check.md. It ran before the rebase on main and was not run again.
  • Both collectors and infrahub-backup create were run against the least-privilege PostgreSQL setup of the Infrahub production deployment documentation ("Least-privilege service credentials"): a non-superuser prefect role that owns the prefect database and the public schema, on postgres:18-alpine started with --auth-host=scram-sha-256, with the Infrahub 1.12.0 task manager connected as that role and a percent-encoded password holding @, a space, /, $ and #. Both storage collectors were success, PostgreSQL's connection log showed every session as user=prefect database=prefect application_name=infrahub-collect, pg_dump connected as prefect as well, and a wrong password in the operator's environment was reported as login rejected.
  • The extraction commit 2380139 changes only Markdown, two Go comments and comments in tests/helpers/bundle.py. go vet, the unit tests, rumdl and ruff pass on it.

Impact & rollout

  • Backward compatibility:
    • Every bundle holds a new file, task-manager-db/storage.json, and the manifest lists two new collectors.
    • The backup format is unchanged.
    • infrahub-backup no longer takes missing parts of the Prefect URL from PG* variables or ~/.pgpass.
  • Both backends: Docker Compose and Kubernetes behave the same. On Kubernetes, the queries run in the primary pod of a CloudNativePG cluster.
  • Config/env changes:
    • New flag --include-task-manager-storage, with the variable INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.
    • INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT lowers the query limits for the end-to-end tests only, and is not documented for operators.
  • Deployment notes: task-manager-db needs sh and psql, which the official postgres, Bitnami and CloudNativePG images include. Without them, the manifest records the collector as failed, and the bundle is still produced.

Checklist

  • Tests added/updated
  • Changelog entry added (changelog/181.added.md, changelog/181.changed.md)
  • External docs updated (docs/docs/collect/, docs/docs/reference/)
  • Internal .md docs updated (dev/adr/0008-*, dev/adr/0009-*, dev/guidelines/collectors.md, dev/knowledge/infrahub-collect.md, AGENTS.md)
  • I have reviewed AI generated content

🤖 Generated with Claude Code


Summary by cubic

Troubleshooting bundles now include the storage use of the task manager database, so support can answer how much disk the database uses and which records fill it without sending the customer SQL queries (closes #181).

Task manager database storage collectors

  • Every bundle now holds task-manager-db/storage.json: database and per-table sizes, live and dead row estimates, and last vacuum times.
  • --include-task-manager-storage (or INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE) adds task-manager-db/storage-breakdown.json: exact row counts and oldest/newest timestamps per record group for the flow run, task run, state, log, event and artifact tables.
  • Queries run read-only in task-manager-db with time limits PostgreSQL enforces; a failed or missing table is recorded in the manifest without stopping the bundle, and the password reaches psql on stdin, not in a process argument.
  • Do not request the breakdown during a task manager upgrade: each query holds an access-share lock on the tables it reads and joins for up to 5 minutes.

infrahub-backup credential lookup

  • The backup tool now reads the task manager database user, password and database only from the Prefect connection URL, taking defaults for missing parts; it no longer falls back to PG* variables, ~/.pgpass or the OS user, and pgx is replaced with net/url parsing.

Written for commit 419cd9f. Summary will update on new commits.

View guided diff Turn on auto-fix

fatih-acar and others added 27 commits October 26, 2026 17:24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Research verified against a postgres:18-alpine container: PGOPTIONS
enforces the read-only session and statement_timeout, a killed client
leaves the query running until the limit, and turning off hash and merge
joins keeps the breakdown free of temporary files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The time-limit e2e test held a lock that also blocks the always-on
storage query, because the size functions take ACCESS SHARE on every
table. The test variable now lowers both limits and the test asserts
both outcomes. Also add the statistics reset time, a credential fallback
to the task-manager-db environment, and the schema-change edge case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also correct the FR-014 test note: no test covers findPrimaryPod on
main, although the PRD says one exists, so the tasks add it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#181)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the env-output lookup and the connection-string parsing out of
fetchPostgresCredentials and applyPrefectConnection into the pure
functions prefectConnectionFromEnv and parsePrefectConnection, so the
collect path can reuse them without touching iops.config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the task manager database constants and
resolveTaskManagerDBCredentials, which looks up the database login once
per run and caches it on the collect context. It tries the operator's
environment, then the task-manager environment, then the
task-manager-db environment (so the figures stay available when the
Prefect server is down), then the defaults. Both execs are bounded by
the collect exec timeout. Only the source is logged, never a value, and
the env output never reaches the bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#181)

Add the query runner that both storage collectors share:

- buildPsqlCommand runs psql inside task-manager-db with the password in
  the environment only, and a PGOPTIONS that makes PostgreSQL enforce a
  read-only transaction and the time limit. Breakdown queries also turn
  off hash joins, merge joins and parallel workers so they never write
  temporary files. A sub-millisecond limit rounds up, never to 0, which
  PostgreSQL reads as no limit.
- effectiveDBQueryLimit lets INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT lower
  a limit for the end-to-end tests, never raise it.
- classifyDBQueryFailure sorts a failed exec into timed out, psql
  missing, login rejected, not reachable or query failed, and picks the
  line that names the cause, skipping the notices kubectl writes.
- runTaskManagerDBQuery runs the command through the bounded
  ExecSeparateContext at the limit plus 15 seconds, and masks the
  password in the returned command, streams and error text.
- skipWhenTaskManagerDBAbsent skips with the external-PostgreSQL reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No test covered findPrimaryPod. With a CloudNativePG cluster behind
task-manager-db, the storage queries must run on the primary pod,
because a standby does not run VACUUM or autovacuum (FR-014). Cover the
cnpg.io/instanceRole label, the legacy role label, the fallback to the
first pod, a single pod, and the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sked (#181)

runTaskManagerDBQuery masked the password in stdout on success too. The
default password is "prefect", so masking the query result turned the
database name into "********" and would corrupt event names such as
prefect.flow-run.Completed in the breakdown. A successful query's stdout
holds only the query result, never the command line, so it cannot carry
the password from the command.

The command and stderr stay masked in every case, and stdout stays
masked when the query fails, because it then goes to the .err.txt
diagnostics (FR-017).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…181)

Add the task-manager-db-storage collector, which runs right after
task-manager-state and writes task-manager-db/storage.json: the database
size, when PostgreSQL last reset its statistics, and for every table its
data, index and TOAST sizes, the live and dead row estimates and the
last vacuum and autovacuum times (FR-001 to FR-003).

- storageQuery reads all figures in one read-only statement from catalog
  functions and pg_stat_user_tables, under the 60-second limit that
  PostgreSQL enforces.
- parseStorageReport computes the data size as the table size minus
  TOAST and the total as the sum of the parts, so data + indexes +
  TOAST = total holds even when a table grows between two size calls.
  Timestamps are RFC 3339 UTC, a database without tables gives an empty
  list, and tables are ordered by total size, then schema and name.
- A failed query, or output that is not the expected document, writes
  storage.err.txt with the password masked and records the classified
  reason; the collector is skipped with the external-PostgreSQL reason
  when task-manager-db has no replica (FR-010, FR-011, FR-017).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add task-manager-db-storage to the expected collector set and its
task-manager-db/ directory to the bundle layout, and add
assert_storage_report, which checks storage.json against
contracts/storage.schema.json by hand: required fields and no others,
the readable-size pattern, RFC 3339 or null timestamps, data + index +
TOAST = total, and the table order.

The Docker default-run test now runs with --log-format json and checks
that the storage collector succeeds, that storage.json is valid, that
its database size is within 1 MiB of pg_database_size read in
task-manager-db (FR-001), and that the collector takes under 10 seconds
(SC-002). The owner and database come from the container's POSTGRES_USER
and POSTGRES_DB. The breakdown collector's "not requested" outcome is
asserted once that collector exists (T031/T033).

The Kubernetes test checks that storage.json is collected and valid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the create flag and its INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE
variable, read into CollectOptions.IncludeTaskManagerStorage. The flag
registration and option resolution move out of main() so a unit test
can check the flag, the variable and their precedence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ind (#181)

With --include-task-manager-storage, the task-manager-db-storage-breakdown
collector writes task-manager-db/storage-breakdown.json: for events,
flow_run, task_run, flow_run_state, task_run_state, log and artifact,
the exact row count, the oldest and newest timestamp and, for the run
tables, the rows without start_time, per group of records.

A catalog query first finds the tables the Prefect version has, so a
missing table is recorded as absent; when it fails, its diagnostics go
to storage-breakdown.err.txt and no table is read. Each table then runs
in its own read-only query, bounded at 5 minutes by PostgreSQL, with
the planner settings that keep joins off temporary files. A failed
table keeps the others: the manifest reason names each failed table and
its cause, so the per-table timeout is deliberately not wrapped.

The collector runs first among the opt-in collectors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181)

Add task-manager-db-storage-breakdown to the expected and opt-in collector
sets, and assert_storage_breakdown, which checks storage-breakdown.json
against its contract by hand.

On Docker Compose, seed flow runs through the Prefect client in a
task-worker container, collect the breakdown with the flag and with the
environment variable, and check the seeded counts, including the flow run
cancelled before it started. Lock the artifact table in a separate session
and check that PostgreSQL stops both the storage query and the artifact
breakdown at the lowered limit, and that no tool session is left. The
default bundle test now checks that the breakdown is skipped as not
requested.

The lock is held for up to 300 seconds instead of 120: the lock also
blocks `prefect flow-run ls`, so task-manager-state waits for its own
60-second limit and a run takes about 85 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe task-manager-db/storage.json (every bundle) and
task-manager-db/storage-breakdown.json (--include-task-manager-storage)
in the bundle contents, add the flag and INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE
to the command and configuration references, and add a how-to section
on the breakdown: the 5-minute limit per table, failed and absent
tables, not using the flag during a task manager upgrade, the 60-second
storage figures that an upgrade can stop, and how to stop the queries
with pg_terminate_backend by application name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ries (#181)

Add ADR 0008: queries in the task manager database run in read-only
sessions with time limits that PostgreSQL enforces through PGOPTIONS,
backed by a tool-side exec bound 15 seconds longer, and the breakdown
turns off hash joins, merge joins and parallel workers so it writes no
temporary files. The ADR also records the value-based password masking
of the query runner and why a successful query's stdout is not masked.

Add the database query rules and the named sub-result exception for
timeout wrapping to the collector guidelines, describe the two storage
collectors, the credential lookup order, the lock behavior of the size
functions and the test-only time limit variable in the infrahub-collect
knowledge page, and list the new files in AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Tick T045 and T046, and record the CloudNativePG primary-pod check in
t046-cnpg-check.md for the pull request description.

Change the wrong password in quickstart scenario 5 from `x` to
`wrong-secret`: the tool masks every occurrence of the password, so with
`x` the reason read `role "nobody" does not e********ist`. Explain why
the password must differ from the user name, the database name and the
words of the expected output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…181)

A Prefect connection URL that does not parse, in the operator's
environment or in the task-manager container, was ignored with only a
debug message. No flag turns on debug logging, so an operator who set
the URL to fix a rejected login, for example with an unencoded "#" or
"/" in the password, never learned that it was ignored.

Both the collect lookup and the backup tool now log a warning that names
the variable and where it was found, with a hint to percent-encode
special characters in the password. The warning holds no part of the
URL: pgx redacts the password in the URL it quotes, but the cause it
adds can quote part of the password (`invalid port ":pa" after host`),
so no parse cause is logged. The backup tool's warning used to name
PREFECT_API_DATABASE_CONNECTION_URL even when
PREFECT_SERVER_DATABASE_CONNECTION_URL was the variable that failed;
prefectConnectionFromEnv now also returns the variable name.

Also:
- test FR-017 on the two paths where the query succeeds but its output
  does not parse, where only the collector's mask keeps the password out
  of storage.err.txt and storage-breakdown.err.txt;
- correct comments on iops.config, on where the password travels, on
  how pgx fills a part the URL leaves out, and on which manifest reasons
  start with the error kind;
- correct the CNPG check record (findPrimaryPod also reads the legacy
  role label) and the assert_storage_breakdown docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y default (#181)

When task-manager-db sets POSTGRES_USER and POSTGRES_PASSWORD but
neither POSTGRES_DB nor POSTGRES_DATABASE, the lookup used the database
`prefect`. The official postgres image, which Infrahub's Docker Compose
deployment uses for task-manager-db, names the database after
POSTGRES_USER in that case, so the lookup now does the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each breakdown query takes an ACCESS SHARE lock on the table it reads
and on the tables it joins: flow for flow_run, and flow_run and flow for
task_run and log. A schema change on flow_run therefore makes three
breakdown tables wait and time out, not one. While a schema change waits
behind a breakdown query, PostgreSQL also makes later queries on that
table, including the task manager's, wait behind the schema change. The
size functions wait only for an ACCESS EXCLUSIVE lock, the one mode that
conflicts with ACCESS SHARE, not for any lock.

pg_terminate_backend can end another role's session only for a
superuser, a member of that role, or a role with the privileges of
pg_signal_backend, and only a superuser can end a superuser's session.
The page now says to run the statement as a superuser or as the role
infrahub-collect logged in with, names where that role comes from, and
says when the examples ask for a password.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age (#181)

Record the chunk results, the local-pass evidence of every new or
modified test, the review findings with their status, the decisions made
during implementation, and the next steps before a pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it (#181)

The branch now sits on origin/main at fa67557, which contains the pod
lookup fixes f673202 and 1e0e522. The CloudNativePG check and the
implementation report still said the branch lacked them, and the report
asked for the rebase as its first next step.

The report now maps each commit to its SHA after the rebase, describes
the two conflicts and how both sides were kept, and lists the Docker and
Kubernetes end-to-end runs on 6f21171 as the latest passing runs. The
CloudNativePG check says which of its runs came before the rebase and
that it was not run again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The storage collector runs in every `infrahub-collect create`, and it
put the task manager database password in the arguments of the docker
or kubectl process, where other users of the host can read it in the
process list. psql now runs through a short sh script that reads the
password from the first line of stdin into PGPASSWORD. Both backends
gain ExecSeparateInputContext, a bounded exec with stdin: docker compose
exec forwards stdin, and kubectl exec does with -i. A missing psql is
now reported by sh as "psql: not found".

Other fixes from the review of the branch:

- The task-manager-db lookup uses POSTGRES_USER or POSTGRES_PASSWORD on
  its own. A missing user becomes postgres and a missing password takes
  the default, instead of the value that was found being discarded.
- A password equal to the user or the database name is no longer
  masked, so the reason of a failed query with the default login still
  names the database.
- The Prefect connection URL is parsed with url.Parse instead of
  pgx.ParseConfig, which filled missing parts from PGUSER, PGPASSWORD,
  PGDATABASE, ~/.pgpass, PGSERVICE and the operating-system user of the
  machine that runs the tool. This also applies to infrahub-backup, and
  pgx is no longer a dependency. The warning about a URL that does not
  parse names the part that failed, without quoting the URL.
- The --include-task-manager-storage help text names the tables the
  breakdown reads, instead of "its largest tables".
- The breakdown catalog query is built from breakdownTables, the query
  time limit variable is read once per run, cmp.Or replaces
  valueOrDefault, and one helper reads a variable from env output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181)

ADR 0008 now describes the password on stdin, the sh requirement in
task-manager-db, and why a password equal to the user or the database
name is not masked. It drops the known limitation about the process
list, which only `infrahub-backup create` still has, and adds the
`docker compose exec -e PGPASSWORD` alternative and why it was not
chosen.

The collector guidelines and the knowledge page describe
ExecSeparateInputContext, cc.dbQueryLimit and passwordMask, the
task-manager-db fallbacks when only the user or only the password is
set, and the parsing of the Prefect connection URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fatih-acar fatih-acar added enhancement New feature or request go Go code changes backup Backup tool changes labels Oct 27, 2026
@fatih-acar fatih-acar added the changes/minor Release with new features label Oct 27, 2026
@fatih-acar
fatih-acar marked this pull request as ready for review November 12, 2026 16:15
@fatih-acar
fatih-acar requested a review from a team as a code owner November 12, 2026 16:15

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

34 issues found across 48 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="specs/006-taskmgr-db-storage/contracts/cli.md">

<violation number="1" location="specs/006-taskmgr-db-storage/contracts/cli.md:13">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

`infrahub-backup` does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous `PG*`/`.pgpass` fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/plan.md">

<violation number="1" location="specs/006-taskmgr-db-storage/plan.md:14">
P2: Disabling join methods does not guarantee that PostgreSQL will never write temporary files: the grouped breakdown subquery can still spill through hash aggregation, sorting, or materialization when many distinct flow names exist. Add a hard database-level safeguard or qualify this guarantee and the SC-005 mitigation.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/plan.md:20">
P3: This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.</violation>

<violation number="3" location="specs/006-taskmgr-db-storage/plan.md:114">
P2: The plan records `runTaskManagerDBQuery` as returning only `string`, although failed queries need the command, stdout, and stderr for `writeDumpFailure` and T010 defines `dbQueryResult`. Record the result type here so the plan describes the diagnostics contract.</violation>

<violation number="4" location="specs/006-taskmgr-db-storage/plan.md:116">
P3: The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.</violation>

<violation number="5" location="specs/006-taskmgr-db-storage/plan.md:147">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

This plan still describes the password as a `docker`/`kubectl` argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says `pgx` remains and the module files/vendor hash are unchanged.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/research.md">

<violation number="1" location="specs/006-taskmgr-db-storage/research.md:49">
P1: Disabling hash and merge joins does not prevent the `GROUP BY` aggregate from spilling to `base/pgsql_tmp` when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/research.md:99">
P2: R8 says the database-container fallback requires both `POSTGRES_USER` and `POSTGRES_PASSWORD`, but `credentialsFromPostgresEnv` activates when either is set and defaults the missing value. Document the actual fallback behavior.</violation>

<violation number="3" location="specs/006-taskmgr-db-storage/research.md:127">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

R10 says the feature exposes the password in `docker`/`kubectl` arguments and retains that behavior, but the runner sends it on stdin through `ExecSeparateInputContext`. Update this claim and R1’s stale `PGPASSWORD` description to match the shipped implementation.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/tasks.md">

<violation number="1" location="specs/006-taskmgr-db-storage/tasks.md:30">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

This instruction conflicts with the changes in this PR: `pgx/v5` and its related entries were removed from `go.mod`/`go.sum`, and `flake.nix`'s `vendorHash` changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/tasks.md:48">
P3: T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than `ok`. Update this task and its test references so the checklist remains usable.</violation>

<violation number="3" location="specs/006-taskmgr-db-storage/tasks.md:50">
P2: The credential-source rule is contradictory: T004 says both `POSTGRES_USER` and `POSTGRES_PASSWORD` are required, but T005 describes a user-only case falling through, while the implementation accepts partial values. Choose one fallback policy and align the task, research, tests, and resolver before treating this phase as complete.</violation>

<violation number="4" location="specs/006-taskmgr-db-storage/tasks.md:52">
P2: T006 specifies `PGPASSWORD` in the container environment and `ExecSeparateContext`, but ADR 0008 and the runner use stdin with `ExecSeparateInputContext`. Update T006, T010, and T011 to the stdin contract so the completed task list does not prescribe a password-bearing exec interface.</violation>

<violation number="5" location="specs/006-taskmgr-db-storage/tasks.md:56">
P2: T010 says both returned streams are always masked, but successful stdout is the JSON that collectors parse. With the default password `prefect`, value masking would rewrite database and event names and corrupt successful reports; mask stdout only on failure, as ADR 0008 specifies.</violation>
</file>

<file name="src/internal/app/app_config.go">

<violation number="1" location="src/internal/app/app_config.go:241">
P2: This scheme check accepts malformed PostgreSQL URLs such as `postgresql+asyncpg:malformed` and unrelated schemes such as `postgresqlx://...`, so invalid connection settings can be silently ignored or applied instead of producing the promised warning. Require a hierarchical URL and allow only `postgres`, `postgresql`, or a supported `postgresql+...` driver scheme.</violation>
</file>

<file name="docs/docs/collect/index.mdx">

<violation number="1" location="docs/docs/collect/index.mdx:40">
P3: `storage.json` is not present in every bundle: the collector is skipped when `task-manager-db` is absent and writes `storage.err.txt` on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/quickstart.md">

<violation number="1" location="specs/006-taskmgr-db-storage/quickstart.md:33">
P2: This extraction command fails on a fresh checkout because `./out` does not exist. Create the extraction directory before running `tar`.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/quickstart.md:58">
P2: The breakdown validation reads an archive that these commands never extract, so `jq` fails or inspects the previous default bundle. Use separate output directories and extract the selected archive before reading `storage-breakdown.json`.</violation>

<violation number="3" location="specs/006-taskmgr-db-storage/quickstart.md:105">
P2: This validation describes the equal-password case backwards: the collector deliberately leaves that value visible so the diagnostic retains the role or database name. Document the exception accurately, otherwise the default Compose credentials produce an incorrect masking result.</violation>

<violation number="4" location="specs/006-taskmgr-db-storage/quickstart.md:115">
P2: The quickstart presents non-null `last_autovacuum` as evidence that queries hit the primary, but T046 observed zero values after a switchover and says the check cannot identify the answering pod. Use the logged `kubectl exec` target as the verification.</violation>
</file>

<file name="docs/docs/reference/commands.mdx">

<violation number="1" location="docs/docs/reference/commands.mdx:371">
P2: `task-manager-db-storage` is skipped when no `task-manager-db` replica exists, including external PostgreSQL deployments, so “Collected in every run” is inaccurate. Qualify this as applying when the service is deployed and the query succeeds.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json">

<violation number="1" location="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json:13">
P2: This schema accepts an empty or truncated `tables` array even though every breakdown must contain one outcome for each of the seven tables. Require the complete seven-entry list (at minimum `minItems`/`maxItems`, and preferably per-name uniqueness/order) so schema validation cannot accept a report with missing table results.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json:49">
P2: The schema does not bind a table name to its documented grouping, timestamp, or group fields, so schema-only consumers can accept a report whose metadata describes the wrong records. Add table-specific conditional definitions that require the documented `grouped_by`/`timestamp_column`, require exactly those group keys, and prohibit `rows_without_start_time` outside the two run tables.</violation>
</file>

<file name="docs/docs/collect/create.mdx">

<violation number="1" location="docs/docs/collect/create.mdx:232">
P2: This says every bundle contains `storage.json`, but deployments with an external PostgreSQL server have no `task-manager-db` replica and the collector is skipped. Qualify the statement and tell operators to use the manifest outcome when the database service is absent.</violation>
</file>

<file name="dev/adr/0008-database-enforced-query-limits.md">

<violation number="1" location="dev/adr/0008-database-enforced-query-limits.md:20">
P2: This presents the breakdown as avoiding temporary files, but the planner settings only prevent hash/merge-join spills; the `GROUP BY` can still spill when a deployment has many distinct flow names. Qualify this as an observed result for the benchmark data rather than a guarantee, or document an aggregation safeguard.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/opsmill-implement-report.md">

<violation number="1" location="specs/006-taskmgr-db-storage/opsmill-implement-report.md:236">
P2: The report presents checks on `6f21171` as checks on the final implementation, but two later commits materially changed the collector and tests. Update the report with those commits and rerun the gates, or clearly label the evidence as a historical snapshot.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/spec.md">

<violation number="1" location="specs/006-taskmgr-db-storage/spec.md:128">
P2: FR-004 and FR-010 prescribe different reasons for the breakdown collector when `task-manager-db` is absent and the flag is not requested. Define the precedence explicitly so the manifest contract and tests do not diverge.</violation>

<violation number="2" location="specs/006-taskmgr-db-storage/spec.md:130">
P2: FR-011 conflicts with the opt-in behavior: an unrequested breakdown must be `skipped` as `not requested`, but this requirement says both collectors must be `failed` when the database is unreachable or lacks `psql`. Limit FR-011 to requested collectors and state that the not-requested precondition takes precedence.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/contracts/manifest-outcomes.md">

<violation number="1" location="specs/006-taskmgr-db-storage/contracts/manifest-outcomes.md:14">
P2: The layout promises `storage.json` on every run, but runs without a `task-manager-db` replica produce no `task-manager-db/` directory at all and record the collector as skipped. Qualify this entry as conditional on the service being deployed so the bundle contract matches the documented skipped outcome.</violation>
</file>

<file name="AGENTS.md">

<violation number="1" location="AGENTS.md:113">
P2: This says `storage.json` is present in every bundle, but the collector skips it when `task-manager-db` has no replica and writes `storage.err.txt` when the query fails. Qualify the claim to describe the successful, locally deployed case.</violation>
</file>

<file name="specs/006-taskmgr-db-storage/critiques/critique-20261026-174500.md">

<violation number="1" location="specs/006-taskmgr-db-storage/critiques/critique-20261026-174500.md:68">
P2: E3 is stale relative to the implementation: the task-manager database query now passes the password on stdin, so it is no longer exposed in the Docker or Kubernetes exec arguments. Remove or rewrite this finding and update its consolidated entry and recommended action; the current text would send maintainers toward a fix that is already shipped.</violation>
</file>

<file name="src/internal/app/collect_taskmanager_db.go">

<violation number="1" location="src/internal/app/collect_taskmanager_db.go:142">
P2: Do not use a partial `task-manager-db` credential tuple: `POSTGRES_USER` without `POSTGRES_PASSWORD` can occur when the image reads `POSTGRES_PASSWORD_FILE`, and the synthesized default password is not the database password. Require both values before returning these credentials, then fall through to the documented defaults.</violation>
</file>

<file name="changelog/181.changed.md">

<violation number="1" location="changelog/181.changed.md:1">
P3: This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.</violation>
</file>

<file name="dev/guidelines/collectors.md">

<violation number="1" location="dev/guidelines/collectors.md:40">
P2: `dbQuery.breakdown` prevents hash/merge join plans, but it does not guarantee that `GROUP BY` or `json_agg` cannot spill temporary files. Please qualify this guidance so future collectors do not treat the setting as a complete temporary-file safeguard.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

- Problem: support investigates the storage of this database most often when its disk is nearly full. A breakdown query that writes large temporary files to the database volume could fill the disk, and PostgreSQL stops when it cannot write. That would break SC-005: the task manager would stop scheduling flow runs.
- Observed with default settings: the `log` breakdown (grouped by the parent flow run's flow name, so it joins `log` to `flow_run` and `flow`) uses a parallel hash join with two extra worker processes. With 300,000 flow runs the hash table fits in memory (22 MB). The hash table grows with `flow_run`, and once it no longer fits in `work_mem` PostgreSQL splits the join into batches written to temporary files under `base/pgsql_tmp`, on the same volume.
- `temp_file_limit` would cap these files, but only a superuser can set it. The probe confirmed that the owner gets `permission denied`.
- Decision: the breakdown queries set `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0` in `PGOPTIONS`. Observed with these settings: the plan is a nested loop over the `flow_run` primary key with Memoize (memory bounded by `work_mem`), followed by a hash aggregate with 61 groups using 48 kB. No temporary file is written. The query takes 1.0 seconds instead of 0.3 seconds and runs in one process.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Disabling hash and merge joins does not prevent the GROUP BY aggregate from spilling to base/pgsql_tmp when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/research.md, line 49:

<comment>Disabling hash and merge joins does not prevent the `GROUP BY` aggregate from spilling to `base/pgsql_tmp` when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.</comment>

<file context>
@@ -0,0 +1,175 @@
+- Problem: support investigates the storage of this database most often when its disk is nearly full. A breakdown query that writes large temporary files to the database volume could fill the disk, and PostgreSQL stops when it cannot write. That would break SC-005: the task manager would stop scheduling flow runs.
+- Observed with default settings: the `log` breakdown (grouped by the parent flow run's flow name, so it joins `log` to `flow_run` and `flow`) uses a parallel hash join with two extra worker processes. With 300,000 flow runs the hash table fits in memory (22 MB). The hash table grows with `flow_run`, and once it no longer fits in `work_mem` PostgreSQL splits the join into batches written to temporary files under `base/pgsql_tmp`, on the same volume.
+- `temp_file_limit` would cap these files, but only a superuser can set it. The probe confirmed that the owner gets `permission denied`.
+- Decision: the breakdown queries set `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0` in `PGOPTIONS`. Observed with these settings: the plan is a nested loop over the `flow_run` primary key with Memoize (memory bounded by `work_mem`), followed by a hash aggregate with 61 groups using 48 kB. No temporary file is written. The query takes 1.0 seconds instead of 0.3 seconds and runs in one process.
+- Rationale: grouping produces few groups (flow names times state types or log levels), so the aggregate stays small. Only the join can grow with the data, and a nested loop over a primary-key index never spills. One process instead of three also keeps the load on the task manager's database lower (SC-005), and the 5-minute limit bounds the slower run time.
+- Known residual effect: a sequential scan sets hint bits on pages whose rows were changed recently, and PostgreSQL may write those pages (with data checksums on, the default in PostgreSQL 18, through the write-ahead log as well). This is ordinary PostgreSQL behavior for any read, limited to recently changed pages, and it does not change any data.
</file context>


- Registration follows `--include-backup` in [`main.go`](../../../src/cmd/infrahub-collect/main.go): a `BoolVar` on `createCmd`, bound with `viper.BindPFlag("include-task-manager-storage", ...)`, read with `viper.GetBool` into `CollectOptions.IncludeTaskManagerStorage`. The flag takes precedence over the environment variable, through the existing `INFRAHUB_` prefix binding in `ConfigureRootCommand`.
- Help text: `Also collect the storage breakdown of the task manager database (reads every row of its run, state, log, event and artifact tables; up to 5 minutes per table)`.
- `infrahub-backup` and `infrahub-taskmanager` do not change.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

infrahub-backup does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous PG*/.pgpass fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/contracts/cli.md, line 13:

<comment>`infrahub-backup` does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous `PG*`/`.pgpass` fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.</comment>

<file context>
@@ -0,0 +1,38 @@
+
+- Registration follows `--include-backup` in [`main.go`](../../../src/cmd/infrahub-collect/main.go): a `BoolVar` on `createCmd`, bound with `viper.BindPFlag("include-task-manager-storage", ...)`, read with `viper.GetBool` into `CollectOptions.IncludeTaskManagerStorage`. The flag takes precedence over the environment variable, through the existing `INFRAHUB_` prefix binding in `ConfigureRootCommand`.
+- Help text: `Also collect the storage breakdown of the task manager database (reads every row of its run, state, log, event and artifact tables; up to 5 minutes per table)`.
+- `infrahub-backup` and `infrahub-taskmanager` do not change.
+
+## No flag for the always-collected figures
</file context>
Suggested change
- `infrahub-backup` and `infrahub-taskmanager` do not change.
- No CLI flag is added to `infrahub-backup` or `infrahub-taskmanager`.

| The password leaks into `.err.txt` through the `env PGPASSWORD=...` prefix on Kubernetes | Value-based masking of every string the runner returns and writes; unit test (FR-017). |
| Credentials cannot be read when the Prefect server is down, which is common when the database has filled its disk | Fall back to the `task-manager-db` container's environment before the defaults (critique X1). If the login still fails, the reason `login rejected: ...` names the role, and the bundle is still produced. |
| A schema change holds a table lock when the storage query runs | The size functions wait for the lock; PostgreSQL stops the query at 60 seconds, and the collector records `timed out after 60s`. The documentation mentions it (critique P4). |
| The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

This plan still describes the password as a docker/kubectl argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says pgx remains and the module files/vendor hash are unchanged.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 147:

<comment>This plan still describes the password as a `docker`/`kubectl` argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says `pgx` remains and the module files/vendor hash are unchanged.</comment>

<file context>
@@ -0,0 +1,152 @@
+| The password leaks into `.err.txt` through the `env PGPASSWORD=...` prefix on Kubernetes | Value-based masking of every string the runner returns and writes; unit test (FR-017). |
+| Credentials cannot be read when the Prefect server is down, which is common when the database has filled its disk | Fall back to the `task-manager-db` container's environment before the defaults (critique X1). If the login still fails, the reason `login rejected: ...` names the role, and the bundle is still produced. |
+| A schema change holds a table lock when the storage query runs | The size functions wait for the lock; PostgreSQL stops the query at 60 seconds, and the collector records `timed out after 60s`. The documentation mentions it (critique P4). |
+| The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. |
+| A future Prefect version renames a table or column | A missing table is `absent`; a missing column fails that table only, with the PostgreSQL message as the reason. |
+
</file context>
Suggested change
| The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. |
| The password reaches `psql` on stdin, not in the `docker` or `kubectl` arguments | The query runner reads it from stdin in `task-manager-db`; no follow-up is needed. |

- Finding: on Kubernetes, `prepareCommand` prefixes the command with `env PGPASSWORD=<password>`, and the failure diagnostics that collectors write to `<file>.err.txt` (`writeDumpFailure`) include the command. A failed storage query would therefore write the password into the bundle.
- Decision: the runner masks the resolved password in every string it returns or writes: the recorded command, both output streams and the error message. It passes a value-based mask to `writeDumpFailure` (`strings.ReplaceAll(s, password, "********")` when the password is not empty), on top of the key-name masking of ADR 0004. A unit test fails a query on a fake backend and checks every staged file and the manifest for the password (FR-017).
- The debug lines in `command_executor.go` also print the full command, but `infrahub-collect` has no flag that turns on debug logging, so they never reach the operator.
- Known limitation, shared with `infrahub-backup create` (critique E3): the password is an argument of the `docker exec` or `kubectl exec` process (`-e PGPASSWORD=...`, or `env PGPASSWORD=...` on Kubernetes). Other users of the operator's host can view it in the process list, and a Kubernetes API server that logs exec request URIs records it in its audit log. The PRD asks for the same approach as `pg_dump`, so the feature keeps it. A follow-up for both tools could pass the password on standard input instead, for example `sh -c 'PGPASSWORD="$(cat)" exec psql ...'`, which needs a bounded exec with standard input in the collect backends.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

R10 says the feature exposes the password in docker/kubectl arguments and retains that behavior, but the runner sends it on stdin through ExecSeparateInputContext. Update this claim and R1’s stale PGPASSWORD description to match the shipped implementation.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/research.md, line 127:

<comment>R10 says the feature exposes the password in `docker`/`kubectl` arguments and retains that behavior, but the runner sends it on stdin through `ExecSeparateInputContext`. Update this claim and R1’s stale `PGPASSWORD` description to match the shipped implementation.</comment>

<file context>
@@ -0,0 +1,175 @@
+- Finding: on Kubernetes, `prepareCommand` prefixes the command with `env PGPASSWORD=<password>`, and the failure diagnostics that collectors write to `<file>.err.txt` (`writeDumpFailure`) include the command. A failed storage query would therefore write the password into the bundle.
+- Decision: the runner masks the resolved password in every string it returns or writes: the recorded command, both output streams and the error message. It passes a value-based mask to `writeDumpFailure` (`strings.ReplaceAll(s, password, "********")` when the password is not empty), on top of the key-name masking of ADR 0004. A unit test fails a query on a fake backend and checks every staged file and the manifest for the password (FR-017).
+- The debug lines in `command_executor.go` also print the full command, but `infrahub-collect` has no flag that turns on debug logging, so they never reach the operator.
+- Known limitation, shared with `infrahub-backup create` (critique E3): the password is an argument of the `docker exec` or `kubectl exec` process (`-e PGPASSWORD=...`, or `env PGPASSWORD=...` on Kubernetes). Other users of the operator's host can view it in the process list, and a Kubernetes API server that logs exec request URIs records it in its audit log. The PRD asks for the same approach as `pg_dump`, so the feature keeps it. A follow-up for both tools could pass the password on standard input instead, for example `sh -c 'PGPASSWORD="$(cat)" exec psql ...'`, which needs a bounded exec with standard input in the collect backends.
+
+## R11. Outcomes in the manifest and in the breakdown file
</file context>

- Run Go gates through `make` (`make test`, `make vet`), never bare `go test ./...`: the Makefile adds the build tags that the local toolchain needs. For a focused run, pass `-tags untested_go_version` yourself.
- On the collect path, use only the bounded exec variants (`ExecSeparateContext`, `execDumpTimeout`), as [`dev/guidelines/collectors.md`](../../dev/guidelines/collectors.md) requires.
- Never write the output of `env` from `task-manager` or `task-manager-db` to the bundle or the log.
- `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

This instruction conflicts with the changes in this PR: pgx/v5 and its related entries were removed from go.mod/go.sum, and flake.nix's vendorHash changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/tasks.md, line 30:

<comment>This instruction conflicts with the changes in this PR: `pgx/v5` and its related entries were removed from `go.mod`/`go.sum`, and `flake.nix`'s `vendorHash` changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.</comment>

<file context>
@@ -0,0 +1,193 @@
+- Run Go gates through `make` (`make test`, `make vet`), never bare `go test ./...`: the Makefile adds the build tags that the local toolchain needs. For a focused run, pass `-tags untested_go_version` yourself.
+- On the collect path, use only the bounded exec variants (`ExecSeparateContext`, `execDumpTimeout`), as [`dev/guidelines/collectors.md`](../../dev/guidelines/collectors.md) requires.
+- Never write the output of `env` from `task-manager` or `task-manager-db` to the bundle or the log.
+- `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module).
+
+---
</file context>
Suggested change
- `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module).
+ `go.mod` and `go.sum` may change to remove unused dependencies; update `flake.nix`'s `vendorHash` when module files change.


**CRITICAL**: no user story work can begin until this phase is complete.

- [X] T002 In `src/internal/app/app_config.go`, extract two pure functions without changing the behavior of `fetchPostgresCredentials` or `applyPrefectConnection`: `prefectConnectionFromEnv(envOutput string) string`, which returns the first non-empty value of `prefectConnectionEnvVars` in preference order from `env` output, and `parsePrefectConnection(connStr string) (user, password, database string, ok bool)`, which holds the normalization and `pgx.ParseConfig` logic now inside `applyPrefectConnection`. Make `fetchPostgresCredentials` and `applyPrefectConnection` call them.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than ok. Update this task and its test references so the checklist remains usable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/tasks.md, line 48:

<comment>T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than `ok`. Update this task and its test references so the checklist remains usable.</comment>

<file context>
@@ -0,0 +1,193 @@
+
+**CRITICAL**: no user story work can begin until this phase is complete.
+
+- [X] T002 In `src/internal/app/app_config.go`, extract two pure functions without changing the behavior of `fetchPostgresCredentials` or `applyPrefectConnection`: `prefectConnectionFromEnv(envOutput string) string`, which returns the first non-empty value of `prefectConnectionEnvVars` in preference order from `env` output, and `parsePrefectConnection(connStr string) (user, password, database string, ok bool)`, which holds the normalization and `pgx.ParseConfig` logic now inside `applyPrefectConnection`. Make `fetchPostgresCredentials` and `applyPrefectConnection` call them.
+- [X] T003 [P] Add table-driven tests for both functions in `src/internal/app/app_config_test.go`: the Prefect 3 variable wins over the Prefect 2 variable whatever the line order; no variable gives an empty string; `postgresql+asyncpg://prefect:secret@task-manager-db:5432/prefect` parses to `prefect`, `secret`, `prefect`; an unparsable string gives `ok == false`.
+- [X] T004 Create `src/internal/app/collect_taskmanager_db.go` with the constants and credentials: `taskManagerDBService = "task-manager-db"`, `taskManagerDBBundleDir = "task-manager-db"`, `taskManagerDBAppName = "infrahub-collect"`, `collectDBQueryTimeout = 5 * time.Minute` (each breakdown table), `collectDBQueryGrace = 15 * time.Second`, `dbQueryTimeoutEnvVar = "INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT"`, and `taskManagerDBNotDeployedReason = "service not deployed; the task manager may use an external PostgreSQL server"`. Add a `taskManagerDBCredentials` struct (user, password, database, source) and `resolveTaskManagerDBCredentials(cc *collectContext) taskManagerDBCredentials`, which tries, in order (research R8): (1) `prefectConnectionEnvVars` in the operator's environment through `os.Getenv`; (2) `env` in `task-manager` through `cc.execDumpTimeout(collectExecTimeout, "task-manager", []string{"env"})` and `prefectConnectionFromEnv`; (3) `env` in `task-manager-db` through the same bounded exec, using `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` or else `POSTGRES_DATABASE`, only when user and password are both present; (4) `defaultPostgresUsername`, `defaultPostgresPassword`, `defaultPostgresDatabase`. Add a `dbCredentials *taskManagerDBCredentials` field to `collectContext` in `src/internal/app/collect.go` and cache the result there, so a run resolves credentials once. Log the source at debug level, never a value. Do not change `iops.config`.
</file context>


The two task manager database collectors write to the `task-manager-db/` directory of the bundle:

- `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: storage.json is not present in every bundle: the collector is skipped when task-manager-db is absent and writes storage.err.txt on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/collect/index.mdx, line 40:

<comment>`storage.json` is not present in every bundle: the collector is skipped when `task-manager-db` is absent and writes `storage.err.txt` on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.</comment>

<file context>
@@ -26,11 +28,22 @@ Each bundle is a single `tar.gz` archive. Inside, every file lives under a top-l
 
+The two task manager database collectors write to the `task-manager-db/` directory of the bundle:
+
+- `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle.
+- `task-manager-db/storage-breakdown.json` holds the storage breakdown. It is in the bundle only when you run `infrahub-collect create` with `--include-task-manager-storage`.
+
</file context>
Suggested change
- `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle.
- `task-manager-db/storage.json` holds the database size and the figures for each table when the collector succeeds; otherwise, the manifest records a skipped or failed collector.

1. `resolveTaskManagerDBCredentials(cc)` runs once per run and caches the result on the collect context: operator environment, then a bounded `env` exec in `task-manager`, then a bounded `env` exec in `task-manager-db` itself, then defaults (research R8). The third step keeps the figures available when the Prefect server is down.
2. `runTaskManagerDBQuery(cc, query dbQuery) (string, error)` builds:
- the command `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <sql>`
- the environment `PGPASSWORD`, `PGAPPNAME=infrahub-collect`, `PGCONNECT_TIMEOUT=10`, and `PGOPTIONS` with `default_transaction_read_only=on`, `statement_timeout`, `TimeZone=UTC`, and, for the breakdown, `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 116:

<comment>The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.</comment>

<file context>
@@ -0,0 +1,152 @@
+1. `resolveTaskManagerDBCredentials(cc)` runs once per run and caches the result on the collect context: operator environment, then a bounded `env` exec in `task-manager`, then a bounded `env` exec in `task-manager-db` itself, then defaults (research R8). The third step keeps the figures available when the Prefect server is down.
+2. `runTaskManagerDBQuery(cc, query dbQuery) (string, error)` builds:
+   - the command `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <sql>`
+   - the environment `PGPASSWORD`, `PGAPPNAME=infrahub-collect`, `PGCONNECT_TIMEOUT=10`, and `PGOPTIONS` with `default_transaction_read_only=on`, `statement_timeout`, `TimeZone=UTC`, and, for the breakdown, `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0`
+   - It runs the command with `ExecSeparateContext` on `task-manager-db`, bounded at the PostgreSQL limit plus 15 seconds, and returns stdout or a `*dbQueryError` (research R3, R9).
+   - The undocumented variable `INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT` can lower both limits for the end-to-end test, never raise them ([contracts/cli.md](contracts/cli.md)).
</file context>


**Language/Version**: Go 1.25.0 (pinned in `go.mod`)

**Primary Dependencies**: no new dependency. Uses cobra, viper, logrus, and `pgx` (already used only to parse the connection URL). Deployment access stays through the `docker` and `kubectl` command-line tools (ADR 0001).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 20:

<comment>This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.</comment>

<file context>
@@ -0,0 +1,152 @@
+
+**Language/Version**: Go 1.25.0 (pinned in `go.mod`)
+
+**Primary Dependencies**: no new dependency. Uses cobra, viper, logrus, and `pgx` (already used only to parse the connection URL). Deployment access stays through the `docker` and `kubectl` command-line tools (ADR 0001).
+
+**Storage**: none. The tool reads the task manager's PostgreSQL database (Prefect 3 schema) in read-only sessions and writes two JSON files into the bundle staging directory.
</file context>

Comment thread changelog/181.changed.md
@@ -0,0 +1 @@
`infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At changelog/181.changed.md, line 1:

<comment>This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.</comment>

<file context>
@@ -0,0 +1 @@
+`infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL.
</file context>
Suggested change
`infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL.
`infrahub-backup` now resolves task manager database credentials from the Prefect connection URL in the operator environment, then the task-manager environment, and finally the defaults; it never uses `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user for omitted URL parts. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL.

The pull request already wrote most of the spec into dev/: ADR 0008,
the storage section of the collect knowledge page and the
timeout-wrapping exception in the collector guidelines. This extraction
adds what was still missing and archives the spec.

ADR 0009 records that PostgreSQL is reached only through psql, pg_dump
and pg_restore inside task-manager-db, over TCP to localhost, with the
alternatives that research R1 rejected: a Go client, which has no
network path through docker exec or kubectl exec; Unix-socket peer
authentication, which the Bitnami image and CloudNativePG refuse for
the owner; and the Prefect server container.

The knowledge page gains the file contracts of storage.json and
storage-breakdown.json, the manifest reason for each outcome, the five
error kinds and the progress lines, and the privileges the queries
need. The non-superuser prefect role from the Infrahub production
deployment documentation was checked against postgres:18-alpine with
password authentication required on localhost: both collectors and
infrahub-backup create log in as that role. A gotcha records that the
official image trusts localhost, so a wrong password still logs in on
Docker Compose.

The collector guidelines gain the one-JSON-value-per-query rule, the
breakdown list as data, and how to test a database time limit end to
end and a change to the password path.

The spec moves to specs/archive/ with its extraction record, and the
eight references to its path in ADR 0008, two Go comments and
tests/helpers/bundle.py point at the archive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Deploying infrahub-ops-cli with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2380139
Status: ✅  Deploy successful!
Preview URL: https://aa7de4ef.infrahub-ops-cli.pages.dev
Branch Preview URL: https://fac-report-taskmgr-storage-u.infrahub-ops-cli.pages.dev

View logs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 existing issues remain and 3 new issues found across 24 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dev/knowledge/infrahub-collect.md">

<violation number="1" location="dev/knowledge/infrahub-collect.md:124">
P2: This outcome is incorrect for Docker deployments with a stopped-but-deployed `task-manager-db`: the collector is not skipped because `ServiceReplicas` includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.</violation>
</file>

<file name="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md">

<violation number="1" location="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md:9">
P3: This rationale is misleading: `docker exec` and `kubectl exec` run the command inside the target container, where `localhost:5432` reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.</violation>

<violation number="2" location="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md:13">
P2: This decision overstates the implementation: `restorePostgreSQL` can use Unix-socket authentication and always targets the `postgres` maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.</violation>
</file>

Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

| Case | `task-manager-db-storage` | `task-manager-db-storage-breakdown` |
|---|---|---|
| Neither the flag nor `INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE` is set | runs anyway | `skipped`, `not requested` |
| `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This outcome is incorrect for Docker deployments with a stopped-but-deployed task-manager-db: the collector is not skipped because ServiceReplicas includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/knowledge/infrahub-collect.md, line 124:

<comment>This outcome is incorrect for Docker deployments with a stopped-but-deployed `task-manager-db`: the collector is not skipped because `ServiceReplicas` includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.</comment>

<file context>
@@ -92,6 +92,59 @@ The size functions (`pg_total_relation_size`, `pg_table_size`, `pg_indexes_size`
+| Case | `task-manager-db-storage` | `task-manager-db-storage-breakdown` |
+|---|---|---|
+| Neither the flag nor `INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE` is set | runs anyway | `skipped`, `not requested` |
+| `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same |
+| The query, or the breakdown's catalog query, reaches its limit | `failed`, `timed out after 60s` | `failed`, `timed out after 60s`; no table is attempted |
+| `psql` is missing, the login is rejected, the server cannot be reached, or another error | `failed`, `psql missing: <cause>`, `login rejected: <cause>`, `not reachable: <cause>` or `query failed: <cause>` | the same, for the catalog query; no table is attempted |
</file context>
Suggested change
| `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same |
| `task-manager-db` has no container | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same |


## Decision

Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This decision overstates the implementation: restorePostgreSQL can use Unix-socket authentication and always targets the postgres maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md, line 13:

<comment>This decision overstates the implementation: `restorePostgreSQL` can use Unix-socket authentication and always targets the `postgres` maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.</comment>

<file context>
@@ -0,0 +1,31 @@
+
+## Decision
+
+Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container.
+
+Queries run through `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <query>`. `-X` ignores a `.psqlrc` file in the container, and `-A -t` prints the bare value without alignment or headers. Each query returns one JSON value that PostgreSQL builds with `json_build_object` and `json_agg`, and the tool parses that JSON. It never splits `psql` text output on delimiters, because flow names and event names can contain any character.
</file context>
Suggested change
Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container.
The `infrahub-collect` storage queries run inside the `task-manager-db` container and connect over TCP to `localhost` as the database owner, using credentials resolved from the Prefect connection URL or documented fallbacks. The URL's host and port are ignored. `infrahub-backup` also runs `pg_dump` and `pg_restore` there, but its restore path may use Unix-socket authentication and the `postgres` maintenance database.


## Context

The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This rationale is misleading: docker exec and kubectl exec run the command inside the target container, where localhost:5432 reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md, line 9:

<comment>This rationale is misleading: `docker exec` and `kubectl exec` run the command inside the target container, where `localhost:5432` reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.</comment>

<file context>
@@ -0,0 +1,31 @@
+
+## Context
+
+The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL.
+
+## Decision
</file context>
Suggested change
The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL.
The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001); those tools execute commands inside `task-manager-db`, where `localhost:5432` reaches PostgreSQL, but they do not expose that port directly to the host. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #181, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backup Backup tool changes changes/minor Release with new features enhancement New feature or request go Go code changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Troubleshooting bundles do not show what uses storage in the task manager's PostgreSQL database

1 participant