Repository navigation
feat(collect): report task manager database storage in bundles - #193
fatih-acar wants to merge 28 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Research verified against a postgres:18-alpine container: PGOPTIONS enforces the read-only session and statement_timeout, a killed client leaves the query running until the limit, and turning off hash and merge joins keeps the breakdown free of temporary files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The time-limit e2e test held a lock that also blocks the always-on storage query, because the size functions take ACCESS SHARE on every table. The test variable now lowers both limits and the test asserts both outcomes. Also add the statistics reset time, a credential fallback to the task-manager-db environment, and the schema-change edge case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also correct the FR-014 test note: no test covers findPrimaryPod on main, although the PRD says one exists, so the tasks add it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#181) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the env-output lookup and the connection-string parsing out of fetchPostgresCredentials and applyPrefectConnection into the pure functions prefectConnectionFromEnv and parsePrefectConnection, so the collect path can reuse them without touching iops.config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the task manager database constants and resolveTaskManagerDBCredentials, which looks up the database login once per run and caches it on the collect context. It tries the operator's environment, then the task-manager environment, then the task-manager-db environment (so the figures stay available when the Prefect server is down), then the defaults. Both execs are bounded by the collect exec timeout. Only the source is logged, never a value, and the env output never reaches the bundle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#181) Add the query runner that both storage collectors share: - buildPsqlCommand runs psql inside task-manager-db with the password in the environment only, and a PGOPTIONS that makes PostgreSQL enforce a read-only transaction and the time limit. Breakdown queries also turn off hash joins, merge joins and parallel workers so they never write temporary files. A sub-millisecond limit rounds up, never to 0, which PostgreSQL reads as no limit. - effectiveDBQueryLimit lets INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT lower a limit for the end-to-end tests, never raise it. - classifyDBQueryFailure sorts a failed exec into timed out, psql missing, login rejected, not reachable or query failed, and picks the line that names the cause, skipping the notices kubectl writes. - runTaskManagerDBQuery runs the command through the bounded ExecSeparateContext at the limit plus 15 seconds, and masks the password in the returned command, streams and error text. - skipWhenTaskManagerDBAbsent skips with the external-PostgreSQL reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No test covered findPrimaryPod. With a CloudNativePG cluster behind task-manager-db, the storage queries must run on the primary pod, because a standby does not run VACUUM or autovacuum (FR-014). Cover the cnpg.io/instanceRole label, the legacy role label, the fallback to the first pod, a single pod, and the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sked (#181) runTaskManagerDBQuery masked the password in stdout on success too. The default password is "prefect", so masking the query result turned the database name into "********" and would corrupt event names such as prefect.flow-run.Completed in the breakdown. A successful query's stdout holds only the query result, never the command line, so it cannot carry the password from the command. The command and stderr stay masked in every case, and stdout stays masked when the query fails, because it then goes to the .err.txt diagnostics (FR-017). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…181) Add the task-manager-db-storage collector, which runs right after task-manager-state and writes task-manager-db/storage.json: the database size, when PostgreSQL last reset its statistics, and for every table its data, index and TOAST sizes, the live and dead row estimates and the last vacuum and autovacuum times (FR-001 to FR-003). - storageQuery reads all figures in one read-only statement from catalog functions and pg_stat_user_tables, under the 60-second limit that PostgreSQL enforces. - parseStorageReport computes the data size as the table size minus TOAST and the total as the sum of the parts, so data + indexes + TOAST = total holds even when a table grows between two size calls. Timestamps are RFC 3339 UTC, a database without tables gives an empty list, and tables are ordered by total size, then schema and name. - A failed query, or output that is not the expected document, writes storage.err.txt with the password masked and records the classified reason; the collector is skipped with the external-PostgreSQL reason when task-manager-db has no replica (FR-010, FR-011, FR-017). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add task-manager-db-storage to the expected collector set and its task-manager-db/ directory to the bundle layout, and add assert_storage_report, which checks storage.json against contracts/storage.schema.json by hand: required fields and no others, the readable-size pattern, RFC 3339 or null timestamps, data + index + TOAST = total, and the table order. The Docker default-run test now runs with --log-format json and checks that the storage collector succeeds, that storage.json is valid, that its database size is within 1 MiB of pg_database_size read in task-manager-db (FR-001), and that the collector takes under 10 seconds (SC-002). The owner and database come from the container's POSTGRES_USER and POSTGRES_DB. The breakdown collector's "not requested" outcome is asserted once that collector exists (T031/T033). The Kubernetes test checks that storage.json is collected and valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the create flag and its INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE variable, read into CollectOptions.IncludeTaskManagerStorage. The flag registration and option resolution move out of main() so a unit test can check the flag, the variable and their precedence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ind (#181) With --include-task-manager-storage, the task-manager-db-storage-breakdown collector writes task-manager-db/storage-breakdown.json: for events, flow_run, task_run, flow_run_state, task_run_state, log and artifact, the exact row count, the oldest and newest timestamp and, for the run tables, the rows without start_time, per group of records. A catalog query first finds the tables the Prefect version has, so a missing table is recorded as absent; when it fails, its diagnostics go to storage-breakdown.err.txt and no table is read. Each table then runs in its own read-only query, bounded at 5 minutes by PostgreSQL, with the planner settings that keep joins off temporary files. A failed table keeps the others: the manifest reason names each failed table and its cause, so the per-table timeout is deliberately not wrapped. The collector runs first among the opt-in collectors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181) Add task-manager-db-storage-breakdown to the expected and opt-in collector sets, and assert_storage_breakdown, which checks storage-breakdown.json against its contract by hand. On Docker Compose, seed flow runs through the Prefect client in a task-worker container, collect the breakdown with the flag and with the environment variable, and check the seeded counts, including the flow run cancelled before it started. Lock the artifact table in a separate session and check that PostgreSQL stops both the storage query and the artifact breakdown at the lowered limit, and that no tool session is left. The default bundle test now checks that the breakdown is skipped as not requested. The lock is held for up to 300 seconds instead of 120: the lock also blocks `prefect flow-run ls`, so task-manager-state waits for its own 60-second limit and a run takes about 85 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe task-manager-db/storage.json (every bundle) and task-manager-db/storage-breakdown.json (--include-task-manager-storage) in the bundle contents, add the flag and INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE to the command and configuration references, and add a how-to section on the breakdown: the 5-minute limit per table, failed and absent tables, not using the flag during a task manager upgrade, the 60-second storage figures that an upgrade can stop, and how to stop the queries with pg_terminate_backend by application name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ries (#181) Add ADR 0008: queries in the task manager database run in read-only sessions with time limits that PostgreSQL enforces through PGOPTIONS, backed by a tool-side exec bound 15 seconds longer, and the breakdown turns off hash joins, merge joins and parallel workers so it writes no temporary files. The ADR also records the value-based password masking of the query runner and why a successful query's stdout is not masked. Add the database query rules and the named sub-result exception for timeout wrapping to the collector guidelines, describe the two storage collectors, the credential lookup order, the lock behavior of the size functions and the test-only time limit variable in the infrahub-collect knowledge page, and list the new files in AGENTS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Tick T045 and T046, and record the CloudNativePG primary-pod check in t046-cnpg-check.md for the pull request description. Change the wrong password in quickstart scenario 5 from `x` to `wrong-secret`: the tool masks every occurrence of the password, so with `x` the reason read `role "nobody" does not e********ist`. Explain why the password must differ from the user name, the database name and the words of the expected output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…181) A Prefect connection URL that does not parse, in the operator's environment or in the task-manager container, was ignored with only a debug message. No flag turns on debug logging, so an operator who set the URL to fix a rejected login, for example with an unencoded "#" or "/" in the password, never learned that it was ignored. Both the collect lookup and the backup tool now log a warning that names the variable and where it was found, with a hint to percent-encode special characters in the password. The warning holds no part of the URL: pgx redacts the password in the URL it quotes, but the cause it adds can quote part of the password (`invalid port ":pa" after host`), so no parse cause is logged. The backup tool's warning used to name PREFECT_API_DATABASE_CONNECTION_URL even when PREFECT_SERVER_DATABASE_CONNECTION_URL was the variable that failed; prefectConnectionFromEnv now also returns the variable name. Also: - test FR-017 on the two paths where the query succeeds but its output does not parse, where only the collector's mask keeps the password out of storage.err.txt and storage-breakdown.err.txt; - correct comments on iops.config, on where the password travels, on how pgx fills a part the URL leaves out, and on which manifest reasons start with the error kind; - correct the CNPG check record (findPrimaryPod also reads the legacy role label) and the assert_storage_breakdown docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y default (#181) When task-manager-db sets POSTGRES_USER and POSTGRES_PASSWORD but neither POSTGRES_DB nor POSTGRES_DATABASE, the lookup used the database `prefect`. The official postgres image, which Infrahub's Docker Compose deployment uses for task-manager-db, names the database after POSTGRES_USER in that case, so the lookup now does the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each breakdown query takes an ACCESS SHARE lock on the table it reads and on the tables it joins: flow for flow_run, and flow_run and flow for task_run and log. A schema change on flow_run therefore makes three breakdown tables wait and time out, not one. While a schema change waits behind a breakdown query, PostgreSQL also makes later queries on that table, including the task manager's, wait behind the schema change. The size functions wait only for an ACCESS EXCLUSIVE lock, the one mode that conflicts with ACCESS SHARE, not for any lock. pg_terminate_backend can end another role's session only for a superuser, a member of that role, or a role with the privileges of pg_signal_backend, and only a superuser can end a superuser's session. The page now says to run the statement as a superuser or as the role infrahub-collect logged in with, names where that role comes from, and says when the examples ask for a password. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age (#181) Record the chunk results, the local-pass evidence of every new or modified test, the review findings with their status, the decisions made during implementation, and the next steps before a pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it (#181) The branch now sits on origin/main at fa67557, which contains the pod lookup fixes f673202 and 1e0e522. The CloudNativePG check and the implementation report still said the branch lacked them, and the report asked for the rebase as its first next step. The report now maps each commit to its SHA after the rebase, describes the two conflicts and how both sides were kept, and lists the Docker and Kubernetes end-to-end runs on 6f21171 as the latest passing runs. The CloudNativePG check says which of its runs came before the rebase and that it was not run again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The storage collector runs in every `infrahub-collect create`, and it put the task manager database password in the arguments of the docker or kubectl process, where other users of the host can read it in the process list. psql now runs through a short sh script that reads the password from the first line of stdin into PGPASSWORD. Both backends gain ExecSeparateInputContext, a bounded exec with stdin: docker compose exec forwards stdin, and kubectl exec does with -i. A missing psql is now reported by sh as "psql: not found". Other fixes from the review of the branch: - The task-manager-db lookup uses POSTGRES_USER or POSTGRES_PASSWORD on its own. A missing user becomes postgres and a missing password takes the default, instead of the value that was found being discarded. - A password equal to the user or the database name is no longer masked, so the reason of a failed query with the default login still names the database. - The Prefect connection URL is parsed with url.Parse instead of pgx.ParseConfig, which filled missing parts from PGUSER, PGPASSWORD, PGDATABASE, ~/.pgpass, PGSERVICE and the operating-system user of the machine that runs the tool. This also applies to infrahub-backup, and pgx is no longer a dependency. The warning about a URL that does not parse names the part that failed, without quoting the URL. - The --include-task-manager-storage help text names the tables the breakdown reads, instead of "its largest tables". - The breakdown catalog query is built from breakdownTables, the query time limit variable is read once per run, cmp.Or replaces valueOrDefault, and one helper reads a variable from env output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#181) ADR 0008 now describes the password on stdin, the sh requirement in task-manager-db, and why a password equal to the user or the database name is not masked. It drops the known limitation about the process list, which only `infrahub-backup create` still has, and adds the `docker compose exec -e PGPASSWORD` alternative and why it was not chosen. The collector guidelines and the knowledge page describe ExecSeparateInputContext, cc.dbQueryLimit and passwordMask, the task-manager-db fallbacks when only the user or only the password is set, and the parsing of the Prefect connection URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
34 issues found across 48 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="specs/006-taskmgr-db-storage/contracts/cli.md">
<violation number="1" location="specs/006-taskmgr-db-storage/contracts/cli.md:13">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
`infrahub-backup` does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous `PG*`/`.pgpass` fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/plan.md">
<violation number="1" location="specs/006-taskmgr-db-storage/plan.md:14">
P2: Disabling join methods does not guarantee that PostgreSQL will never write temporary files: the grouped breakdown subquery can still spill through hash aggregation, sorting, or materialization when many distinct flow names exist. Add a hard database-level safeguard or qualify this guarantee and the SC-005 mitigation.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/plan.md:20">
P3: This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.</violation>
<violation number="3" location="specs/006-taskmgr-db-storage/plan.md:114">
P2: The plan records `runTaskManagerDBQuery` as returning only `string`, although failed queries need the command, stdout, and stderr for `writeDumpFailure` and T010 defines `dbQueryResult`. Record the result type here so the plan describes the diagnostics contract.</violation>
<violation number="4" location="specs/006-taskmgr-db-storage/plan.md:116">
P3: The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.</violation>
<violation number="5" location="specs/006-taskmgr-db-storage/plan.md:147">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
This plan still describes the password as a `docker`/`kubectl` argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says `pgx` remains and the module files/vendor hash are unchanged.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/research.md">
<violation number="1" location="specs/006-taskmgr-db-storage/research.md:49">
P1: Disabling hash and merge joins does not prevent the `GROUP BY` aggregate from spilling to `base/pgsql_tmp` when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/research.md:99">
P2: R8 says the database-container fallback requires both `POSTGRES_USER` and `POSTGRES_PASSWORD`, but `credentialsFromPostgresEnv` activates when either is set and defaults the missing value. Document the actual fallback behavior.</violation>
<violation number="3" location="specs/006-taskmgr-db-storage/research.md:127">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
R10 says the feature exposes the password in `docker`/`kubectl` arguments and retains that behavior, but the runner sends it on stdin through `ExecSeparateInputContext`. Update this claim and R1’s stale `PGPASSWORD` description to match the shipped implementation.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/tasks.md">
<violation number="1" location="specs/006-taskmgr-db-storage/tasks.md:30">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
This instruction conflicts with the changes in this PR: `pgx/v5` and its related entries were removed from `go.mod`/`go.sum`, and `flake.nix`'s `vendorHash` changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/tasks.md:48">
P3: T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than `ok`. Update this task and its test references so the checklist remains usable.</violation>
<violation number="3" location="specs/006-taskmgr-db-storage/tasks.md:50">
P2: The credential-source rule is contradictory: T004 says both `POSTGRES_USER` and `POSTGRES_PASSWORD` are required, but T005 describes a user-only case falling through, while the implementation accepts partial values. Choose one fallback policy and align the task, research, tests, and resolver before treating this phase as complete.</violation>
<violation number="4" location="specs/006-taskmgr-db-storage/tasks.md:52">
P2: T006 specifies `PGPASSWORD` in the container environment and `ExecSeparateContext`, but ADR 0008 and the runner use stdin with `ExecSeparateInputContext`. Update T006, T010, and T011 to the stdin contract so the completed task list does not prescribe a password-bearing exec interface.</violation>
<violation number="5" location="specs/006-taskmgr-db-storage/tasks.md:56">
P2: T010 says both returned streams are always masked, but successful stdout is the JSON that collectors parse. With the default password `prefect`, value masking would rewrite database and event names and corrupt successful reports; mask stdout only on failure, as ADR 0008 specifies.</violation>
</file>
<file name="src/internal/app/app_config.go">
<violation number="1" location="src/internal/app/app_config.go:241">
P2: This scheme check accepts malformed PostgreSQL URLs such as `postgresql+asyncpg:malformed` and unrelated schemes such as `postgresqlx://...`, so invalid connection settings can be silently ignored or applied instead of producing the promised warning. Require a hierarchical URL and allow only `postgres`, `postgresql`, or a supported `postgresql+...` driver scheme.</violation>
</file>
<file name="docs/docs/collect/index.mdx">
<violation number="1" location="docs/docs/collect/index.mdx:40">
P3: `storage.json` is not present in every bundle: the collector is skipped when `task-manager-db` is absent and writes `storage.err.txt` on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/quickstart.md">
<violation number="1" location="specs/006-taskmgr-db-storage/quickstart.md:33">
P2: This extraction command fails on a fresh checkout because `./out` does not exist. Create the extraction directory before running `tar`.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/quickstart.md:58">
P2: The breakdown validation reads an archive that these commands never extract, so `jq` fails or inspects the previous default bundle. Use separate output directories and extract the selected archive before reading `storage-breakdown.json`.</violation>
<violation number="3" location="specs/006-taskmgr-db-storage/quickstart.md:105">
P2: This validation describes the equal-password case backwards: the collector deliberately leaves that value visible so the diagnostic retains the role or database name. Document the exception accurately, otherwise the default Compose credentials produce an incorrect masking result.</violation>
<violation number="4" location="specs/006-taskmgr-db-storage/quickstart.md:115">
P2: The quickstart presents non-null `last_autovacuum` as evidence that queries hit the primary, but T046 observed zero values after a switchover and says the check cannot identify the answering pod. Use the logged `kubectl exec` target as the verification.</violation>
</file>
<file name="docs/docs/reference/commands.mdx">
<violation number="1" location="docs/docs/reference/commands.mdx:371">
P2: `task-manager-db-storage` is skipped when no `task-manager-db` replica exists, including external PostgreSQL deployments, so “Collected in every run” is inaccurate. Qualify this as applying when the service is deployed and the query succeeds.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json">
<violation number="1" location="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json:13">
P2: This schema accepts an empty or truncated `tables` array even though every breakdown must contain one outcome for each of the seven tables. Require the complete seven-entry list (at minimum `minItems`/`maxItems`, and preferably per-name uniqueness/order) so schema validation cannot accept a report with missing table results.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/contracts/storage-breakdown.schema.json:49">
P2: The schema does not bind a table name to its documented grouping, timestamp, or group fields, so schema-only consumers can accept a report whose metadata describes the wrong records. Add table-specific conditional definitions that require the documented `grouped_by`/`timestamp_column`, require exactly those group keys, and prohibit `rows_without_start_time` outside the two run tables.</violation>
</file>
<file name="docs/docs/collect/create.mdx">
<violation number="1" location="docs/docs/collect/create.mdx:232">
P2: This says every bundle contains `storage.json`, but deployments with an external PostgreSQL server have no `task-manager-db` replica and the collector is skipped. Qualify the statement and tell operators to use the manifest outcome when the database service is absent.</violation>
</file>
<file name="dev/adr/0008-database-enforced-query-limits.md">
<violation number="1" location="dev/adr/0008-database-enforced-query-limits.md:20">
P2: This presents the breakdown as avoiding temporary files, but the planner settings only prevent hash/merge-join spills; the `GROUP BY` can still spill when a deployment has many distinct flow names. Qualify this as an observed result for the benchmark data rather than a guarantee, or document an aggregation safeguard.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/opsmill-implement-report.md">
<violation number="1" location="specs/006-taskmgr-db-storage/opsmill-implement-report.md:236">
P2: The report presents checks on `6f21171` as checks on the final implementation, but two later commits materially changed the collector and tests. Update the report with those commits and rerun the gates, or clearly label the evidence as a historical snapshot.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/spec.md">
<violation number="1" location="specs/006-taskmgr-db-storage/spec.md:128">
P2: FR-004 and FR-010 prescribe different reasons for the breakdown collector when `task-manager-db` is absent and the flag is not requested. Define the precedence explicitly so the manifest contract and tests do not diverge.</violation>
<violation number="2" location="specs/006-taskmgr-db-storage/spec.md:130">
P2: FR-011 conflicts with the opt-in behavior: an unrequested breakdown must be `skipped` as `not requested`, but this requirement says both collectors must be `failed` when the database is unreachable or lacks `psql`. Limit FR-011 to requested collectors and state that the not-requested precondition takes precedence.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/contracts/manifest-outcomes.md">
<violation number="1" location="specs/006-taskmgr-db-storage/contracts/manifest-outcomes.md:14">
P2: The layout promises `storage.json` on every run, but runs without a `task-manager-db` replica produce no `task-manager-db/` directory at all and record the collector as skipped. Qualify this entry as conditional on the service being deployed so the bundle contract matches the documented skipped outcome.</violation>
</file>
<file name="AGENTS.md">
<violation number="1" location="AGENTS.md:113">
P2: This says `storage.json` is present in every bundle, but the collector skips it when `task-manager-db` has no replica and writes `storage.err.txt` when the query fails. Qualify the claim to describe the successful, locally deployed case.</violation>
</file>
<file name="specs/006-taskmgr-db-storage/critiques/critique-20261026-174500.md">
<violation number="1" location="specs/006-taskmgr-db-storage/critiques/critique-20261026-174500.md:68">
P2: E3 is stale relative to the implementation: the task-manager database query now passes the password on stdin, so it is no longer exposed in the Docker or Kubernetes exec arguments. Remove or rewrite this finding and update its consolidated entry and recommended action; the current text would send maintainers toward a fix that is already shipped.</violation>
</file>
<file name="src/internal/app/collect_taskmanager_db.go">
<violation number="1" location="src/internal/app/collect_taskmanager_db.go:142">
P2: Do not use a partial `task-manager-db` credential tuple: `POSTGRES_USER` without `POSTGRES_PASSWORD` can occur when the image reads `POSTGRES_PASSWORD_FILE`, and the synthesized default password is not the database password. Require both values before returning these credentials, then fall through to the documented defaults.</violation>
</file>
<file name="changelog/181.changed.md">
<violation number="1" location="changelog/181.changed.md:1">
P3: This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.</violation>
</file>
<file name="dev/guidelines/collectors.md">
<violation number="1" location="dev/guidelines/collectors.md:40">
P2: `dbQuery.breakdown` prevents hash/merge join plans, but it does not guarantee that `GROUP BY` or `json_agg` cannot spill temporary files. Please qualify this guidance so future collectors do not treat the setting as a complete temporary-file safeguard.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| - Problem: support investigates the storage of this database most often when its disk is nearly full. A breakdown query that writes large temporary files to the database volume could fill the disk, and PostgreSQL stops when it cannot write. That would break SC-005: the task manager would stop scheduling flow runs. | ||
| - Observed with default settings: the `log` breakdown (grouped by the parent flow run's flow name, so it joins `log` to `flow_run` and `flow`) uses a parallel hash join with two extra worker processes. With 300,000 flow runs the hash table fits in memory (22 MB). The hash table grows with `flow_run`, and once it no longer fits in `work_mem` PostgreSQL splits the join into batches written to temporary files under `base/pgsql_tmp`, on the same volume. | ||
| - `temp_file_limit` would cap these files, but only a superuser can set it. The probe confirmed that the owner gets `permission denied`. | ||
| - Decision: the breakdown queries set `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0` in `PGOPTIONS`. Observed with these settings: the plan is a nested loop over the `flow_run` primary key with Memoize (memory bounded by `work_mem`), followed by a hash aggregate with 61 groups using 48 kB. No temporary file is written. The query takes 1.0 seconds instead of 0.3 seconds and runs in one process. |
There was a problem hiding this comment.
P1: Disabling hash and merge joins does not prevent the GROUP BY aggregate from spilling to base/pgsql_tmp when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/research.md, line 49:
<comment>Disabling hash and merge joins does not prevent the `GROUP BY` aggregate from spilling to `base/pgsql_tmp` when flow, event, or artifact cardinality is high. The 61-group probe cannot support the stated no-temporary-file guarantee, so this mitigation can still fill a nearly-full database volume; validate high-cardinality data or redesign the aggregation before claiming this safety property.</comment>
<file context>
@@ -0,0 +1,175 @@
+- Problem: support investigates the storage of this database most often when its disk is nearly full. A breakdown query that writes large temporary files to the database volume could fill the disk, and PostgreSQL stops when it cannot write. That would break SC-005: the task manager would stop scheduling flow runs.
+- Observed with default settings: the `log` breakdown (grouped by the parent flow run's flow name, so it joins `log` to `flow_run` and `flow`) uses a parallel hash join with two extra worker processes. With 300,000 flow runs the hash table fits in memory (22 MB). The hash table grows with `flow_run`, and once it no longer fits in `work_mem` PostgreSQL splits the join into batches written to temporary files under `base/pgsql_tmp`, on the same volume.
+- `temp_file_limit` would cap these files, but only a superuser can set it. The probe confirmed that the owner gets `permission denied`.
+- Decision: the breakdown queries set `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0` in `PGOPTIONS`. Observed with these settings: the plan is a nested loop over the `flow_run` primary key with Memoize (memory bounded by `work_mem`), followed by a hash aggregate with 61 groups using 48 kB. No temporary file is written. The query takes 1.0 seconds instead of 0.3 seconds and runs in one process.
+- Rationale: grouping produces few groups (flow names times state types or log levels), so the aggregate stays small. Only the join can grow with the data, and a nested loop over a primary-key index never spills. One process instead of three also keeps the load on the task manager's database lower (SC-005), and the 5-minute limit bounds the slower run time.
+- Known residual effect: a sequential scan sets hint bits on pages whose rows were changed recently, and PostgreSQL may write those pages (with data checksums on, the default in PostgreSQL 18, through the write-ahead log as well). This is ordinary PostgreSQL behavior for any read, limited to recently changed pages, and it does not change any data.
</file context>
|
|
||
| - Registration follows `--include-backup` in [`main.go`](../../../src/cmd/infrahub-collect/main.go): a `BoolVar` on `createCmd`, bound with `viper.BindPFlag("include-task-manager-storage", ...)`, read with `viper.GetBool` into `CollectOptions.IncludeTaskManagerStorage`. The flag takes precedence over the environment variable, through the existing `INFRAHUB_` prefix binding in `ConfigureRootCommand`. | ||
| - Help text: `Also collect the storage breakdown of the task manager database (reads every row of its run, state, log, event and artifact tables; up to 5 minutes per table)`. | ||
| - `infrahub-backup` and `infrahub-taskmanager` do not change. |
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
infrahub-backup does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous PG*/.pgpass fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/contracts/cli.md, line 13:
<comment>`infrahub-backup` does change in this PR: it now takes URL credentials only and defaults omitted parts instead of using the previous `PG*`/`.pgpass` fallbacks. Qualify this as “no CLI flag change” or remove the blanket claim.</comment>
<file context>
@@ -0,0 +1,38 @@
+
+- Registration follows `--include-backup` in [`main.go`](../../../src/cmd/infrahub-collect/main.go): a `BoolVar` on `createCmd`, bound with `viper.BindPFlag("include-task-manager-storage", ...)`, read with `viper.GetBool` into `CollectOptions.IncludeTaskManagerStorage`. The flag takes precedence over the environment variable, through the existing `INFRAHUB_` prefix binding in `ConfigureRootCommand`.
+- Help text: `Also collect the storage breakdown of the task manager database (reads every row of its run, state, log, event and artifact tables; up to 5 minutes per table)`.
+- `infrahub-backup` and `infrahub-taskmanager` do not change.
+
+## No flag for the always-collected figures
</file context>
| - `infrahub-backup` and `infrahub-taskmanager` do not change. | |
| - No CLI flag is added to `infrahub-backup` or `infrahub-taskmanager`. |
| | The password leaks into `.err.txt` through the `env PGPASSWORD=...` prefix on Kubernetes | Value-based masking of every string the runner returns and writes; unit test (FR-017). | | ||
| | Credentials cannot be read when the Prefect server is down, which is common when the database has filled its disk | Fall back to the `task-manager-db` container's environment before the defaults (critique X1). If the login still fails, the reason `login rejected: ...` names the role, and the bundle is still produced. | | ||
| | A schema change holds a table lock when the storage query runs | The size functions wait for the lock; PostgreSQL stops the query at 60 seconds, and the collector records `timed out after 60s`. The documentation mentions it (critique P4). | | ||
| | The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. | |
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
This plan still describes the password as a docker/kubectl argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says pgx remains and the module files/vendor hash are unchanged.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 147:
<comment>This plan still describes the password as a `docker`/`kubectl` argument and proposes a future stdin fix, but the runner now passes it through stdin. Update this resolved risk; the plan also incorrectly says `pgx` remains and the module files/vendor hash are unchanged.</comment>
<file context>
@@ -0,0 +1,152 @@
+| The password leaks into `.err.txt` through the `env PGPASSWORD=...` prefix on Kubernetes | Value-based masking of every string the runner returns and writes; unit test (FR-017). |
+| Credentials cannot be read when the Prefect server is down, which is common when the database has filled its disk | Fall back to the `task-manager-db` container's environment before the defaults (critique X1). If the login still fails, the reason `login rejected: ...` names the role, and the bundle is still produced. |
+| A schema change holds a table lock when the storage query runs | The size functions wait for the lock; PostgreSQL stops the query at 60 seconds, and the collector records `timed out after 60s`. The documentation mentions it (critique P4). |
+| The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. |
+| A future Prefect version renames a table or column | A missing table is `absent`; a missing column fails that table only, with the PostgreSQL message as the reason. |
+
</file context>
| | The password is an argument of the `docker exec` or `kubectl exec` process, as for `pg_dump` in backup | Known limitation shared with backup, recorded in research R10; a follow-up for both tools can pass it on standard input. | | |
| | The password reaches `psql` on stdin, not in the `docker` or `kubectl` arguments | The query runner reads it from stdin in `task-manager-db`; no follow-up is needed. | |
| - Finding: on Kubernetes, `prepareCommand` prefixes the command with `env PGPASSWORD=<password>`, and the failure diagnostics that collectors write to `<file>.err.txt` (`writeDumpFailure`) include the command. A failed storage query would therefore write the password into the bundle. | ||
| - Decision: the runner masks the resolved password in every string it returns or writes: the recorded command, both output streams and the error message. It passes a value-based mask to `writeDumpFailure` (`strings.ReplaceAll(s, password, "********")` when the password is not empty), on top of the key-name masking of ADR 0004. A unit test fails a query on a fake backend and checks every staged file and the manifest for the password (FR-017). | ||
| - The debug lines in `command_executor.go` also print the full command, but `infrahub-collect` has no flag that turns on debug logging, so they never reach the operator. | ||
| - Known limitation, shared with `infrahub-backup create` (critique E3): the password is an argument of the `docker exec` or `kubectl exec` process (`-e PGPASSWORD=...`, or `env PGPASSWORD=...` on Kubernetes). Other users of the operator's host can view it in the process list, and a Kubernetes API server that logs exec request URIs records it in its audit log. The PRD asks for the same approach as `pg_dump`, so the feature keeps it. A follow-up for both tools could pass the password on standard input instead, for example `sh -c 'PGPASSWORD="$(cat)" exec psql ...'`, which needs a bounded exec with standard input in the collect backends. |
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
R10 says the feature exposes the password in docker/kubectl arguments and retains that behavior, but the runner sends it on stdin through ExecSeparateInputContext. Update this claim and R1’s stale PGPASSWORD description to match the shipped implementation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/research.md, line 127:
<comment>R10 says the feature exposes the password in `docker`/`kubectl` arguments and retains that behavior, but the runner sends it on stdin through `ExecSeparateInputContext`. Update this claim and R1’s stale `PGPASSWORD` description to match the shipped implementation.</comment>
<file context>
@@ -0,0 +1,175 @@
+- Finding: on Kubernetes, `prepareCommand` prefixes the command with `env PGPASSWORD=<password>`, and the failure diagnostics that collectors write to `<file>.err.txt` (`writeDumpFailure`) include the command. A failed storage query would therefore write the password into the bundle.
+- Decision: the runner masks the resolved password in every string it returns or writes: the recorded command, both output streams and the error message. It passes a value-based mask to `writeDumpFailure` (`strings.ReplaceAll(s, password, "********")` when the password is not empty), on top of the key-name masking of ADR 0004. A unit test fails a query on a fake backend and checks every staged file and the manifest for the password (FR-017).
+- The debug lines in `command_executor.go` also print the full command, but `infrahub-collect` has no flag that turns on debug logging, so they never reach the operator.
+- Known limitation, shared with `infrahub-backup create` (critique E3): the password is an argument of the `docker exec` or `kubectl exec` process (`-e PGPASSWORD=...`, or `env PGPASSWORD=...` on Kubernetes). Other users of the operator's host can view it in the process list, and a Kubernetes API server that logs exec request URIs records it in its audit log. The PRD asks for the same approach as `pg_dump`, so the feature keeps it. A follow-up for both tools could pass the password on standard input instead, for example `sh -c 'PGPASSWORD="$(cat)" exec psql ...'`, which needs a bounded exec with standard input in the collect backends.
+
+## R11. Outcomes in the manifest and in the breakdown file
</file context>
| - Run Go gates through `make` (`make test`, `make vet`), never bare `go test ./...`: the Makefile adds the build tags that the local toolchain needs. For a focused run, pass `-tags untested_go_version` yourself. | ||
| - On the collect path, use only the bounded exec variants (`ExecSeparateContext`, `execDumpTimeout`), as [`dev/guidelines/collectors.md`](../../dev/guidelines/collectors.md) requires. | ||
| - Never write the output of `env` from `task-manager` or `task-manager-db` to the bundle or the log. | ||
| - `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module). |
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
This instruction conflicts with the changes in this PR: pgx/v5 and its related entries were removed from go.mod/go.sum, and flake.nix's vendorHash changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/tasks.md, line 30:
<comment>This instruction conflicts with the changes in this PR: `pgx/v5` and its related entries were removed from `go.mod`/`go.sum`, and `flake.nix`'s `vendorHash` changed. Update this stale requirement; the checked-off empty-diff claim in T045 is also false.</comment>
<file context>
@@ -0,0 +1,193 @@
+- Run Go gates through `make` (`make test`, `make vet`), never bare `go test ./...`: the Makefile adds the build tags that the local toolchain needs. For a focused run, pass `-tags untested_go_version` yourself.
+- On the collect path, use only the bounded exec variants (`ExecSeparateContext`, `execDumpTimeout`), as [`dev/guidelines/collectors.md`](../../dev/guidelines/collectors.md) requires.
+- Never write the output of `env` from `task-manager` or `task-manager-db` to the bundle or the log.
+- `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module).
+
+---
</file context>
| - `go.mod` and `go.sum` must not change. If they do, the task is wrong: this feature adds no dependency (the logrus test hook `github.com/sirupsen/logrus/hooks/test` is part of the existing logrus module). | |
| + `go.mod` and `go.sum` may change to remove unused dependencies; update `flake.nix`'s `vendorHash` when module files change. |
|
|
||
| **CRITICAL**: no user story work can begin until this phase is complete. | ||
|
|
||
| - [X] T002 In `src/internal/app/app_config.go`, extract two pure functions without changing the behavior of `fetchPostgresCredentials` or `applyPrefectConnection`: `prefectConnectionFromEnv(envOutput string) string`, which returns the first non-empty value of `prefectConnectionEnvVars` in preference order from `env` output, and `parsePrefectConnection(connStr string) (user, password, database string, ok bool)`, which holds the normalization and `pgx.ParseConfig` logic now inside `applyPrefectConnection`. Make `fetchPostgresCredentials` and `applyPrefectConnection` call them. |
There was a problem hiding this comment.
P3: T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than ok. Update this task and its test references so the checklist remains usable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/tasks.md, line 48:
<comment>T002 records parser signatures that the completed implementation does not have: one returns the variable name and connection, and the other returns an error rather than `ok`. Update this task and its test references so the checklist remains usable.</comment>
<file context>
@@ -0,0 +1,193 @@
+
+**CRITICAL**: no user story work can begin until this phase is complete.
+
+- [X] T002 In `src/internal/app/app_config.go`, extract two pure functions without changing the behavior of `fetchPostgresCredentials` or `applyPrefectConnection`: `prefectConnectionFromEnv(envOutput string) string`, which returns the first non-empty value of `prefectConnectionEnvVars` in preference order from `env` output, and `parsePrefectConnection(connStr string) (user, password, database string, ok bool)`, which holds the normalization and `pgx.ParseConfig` logic now inside `applyPrefectConnection`. Make `fetchPostgresCredentials` and `applyPrefectConnection` call them.
+- [X] T003 [P] Add table-driven tests for both functions in `src/internal/app/app_config_test.go`: the Prefect 3 variable wins over the Prefect 2 variable whatever the line order; no variable gives an empty string; `postgresql+asyncpg://prefect:secret@task-manager-db:5432/prefect` parses to `prefect`, `secret`, `prefect`; an unparsable string gives `ok == false`.
+- [X] T004 Create `src/internal/app/collect_taskmanager_db.go` with the constants and credentials: `taskManagerDBService = "task-manager-db"`, `taskManagerDBBundleDir = "task-manager-db"`, `taskManagerDBAppName = "infrahub-collect"`, `collectDBQueryTimeout = 5 * time.Minute` (each breakdown table), `collectDBQueryGrace = 15 * time.Second`, `dbQueryTimeoutEnvVar = "INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT"`, and `taskManagerDBNotDeployedReason = "service not deployed; the task manager may use an external PostgreSQL server"`. Add a `taskManagerDBCredentials` struct (user, password, database, source) and `resolveTaskManagerDBCredentials(cc *collectContext) taskManagerDBCredentials`, which tries, in order (research R8): (1) `prefectConnectionEnvVars` in the operator's environment through `os.Getenv`; (2) `env` in `task-manager` through `cc.execDumpTimeout(collectExecTimeout, "task-manager", []string{"env"})` and `prefectConnectionFromEnv`; (3) `env` in `task-manager-db` through the same bounded exec, using `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` or else `POSTGRES_DATABASE`, only when user and password are both present; (4) `defaultPostgresUsername`, `defaultPostgresPassword`, `defaultPostgresDatabase`. Add a `dbCredentials *taskManagerDBCredentials` field to `collectContext` in `src/internal/app/collect.go` and cache the result there, so a run resolves credentials once. Log the source at debug level, never a value. Do not change `iops.config`.
</file context>
|
|
||
| The two task manager database collectors write to the `task-manager-db/` directory of the bundle: | ||
|
|
||
| - `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle. |
There was a problem hiding this comment.
P3: storage.json is not present in every bundle: the collector is skipped when task-manager-db is absent and writes storage.err.txt on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/collect/index.mdx, line 40:
<comment>`storage.json` is not present in every bundle: the collector is skipped when `task-manager-db` is absent and writes `storage.err.txt` on query failure. Say it is normally present when the collector succeeds, and refer readers to the manifest for skipped or failed outcomes.</comment>
<file context>
@@ -26,11 +28,22 @@ Each bundle is a single `tar.gz` archive. Inside, every file lives under a top-l
+The two task manager database collectors write to the `task-manager-db/` directory of the bundle:
+
+- `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle.
+- `task-manager-db/storage-breakdown.json` holds the storage breakdown. It is in the bundle only when you run `infrahub-collect create` with `--include-task-manager-storage`.
+
</file context>
| - `task-manager-db/storage.json` holds the database size and the figures for each table. It is in every bundle. | |
| - `task-manager-db/storage.json` holds the database size and the figures for each table when the collector succeeds; otherwise, the manifest records a skipped or failed collector. |
| 1. `resolveTaskManagerDBCredentials(cc)` runs once per run and caches the result on the collect context: operator environment, then a bounded `env` exec in `task-manager`, then a bounded `env` exec in `task-manager-db` itself, then defaults (research R8). The third step keeps the figures available when the Prefect server is down. | ||
| 2. `runTaskManagerDBQuery(cc, query dbQuery) (string, error)` builds: | ||
| - the command `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <sql>` | ||
| - the environment `PGPASSWORD`, `PGAPPNAME=infrahub-collect`, `PGCONNECT_TIMEOUT=10`, and `PGOPTIONS` with `default_transaction_read_only=on`, `statement_timeout`, `TimeZone=UTC`, and, for the breakdown, `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0` |
There was a problem hiding this comment.
P3: The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 116:
<comment>The query-runner security description no longer matches the implementation: the password is sent through bounded exec stdin rather than an environment variable or process argument. Update the runner and risk sections so operators and reviewers do not retain the obsolete password-exposure warning.</comment>
<file context>
@@ -0,0 +1,152 @@
+1. `resolveTaskManagerDBCredentials(cc)` runs once per run and caches the result on the collect context: operator environment, then a bounded `env` exec in `task-manager`, then a bounded `env` exec in `task-manager-db` itself, then defaults (research R8). The third step keeps the figures available when the Prefect server is down.
+2. `runTaskManagerDBQuery(cc, query dbQuery) (string, error)` builds:
+ - the command `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <sql>`
+ - the environment `PGPASSWORD`, `PGAPPNAME=infrahub-collect`, `PGCONNECT_TIMEOUT=10`, and `PGOPTIONS` with `default_transaction_read_only=on`, `statement_timeout`, `TimeZone=UTC`, and, for the breakdown, `enable_hashjoin=off`, `enable_mergejoin=off` and `max_parallel_workers_per_gather=0`
+ - It runs the command with `ExecSeparateContext` on `task-manager-db`, bounded at the PostgreSQL limit plus 15 seconds, and returns stdout or a `*dbQueryError` (research R3, R9).
+ - The undocumented variable `INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT` can lower both limits for the end-to-end test, never raise them ([contracts/cli.md](contracts/cli.md)).
</file context>
|
|
||
| **Language/Version**: Go 1.25.0 (pinned in `go.mod`) | ||
|
|
||
| **Primary Dependencies**: no new dependency. Uses cobra, viper, logrus, and `pgx` (already used only to parse the connection URL). Deployment access stays through the `docker` and `kubectl` command-line tools (ADR 0001). |
There was a problem hiding this comment.
P3: This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At specs/006-taskmgr-db-storage/plan.md, line 20:
<comment>This plan is stale about dependency and vendor changes: the implementation removes pgx and updates the Nix vendor hash. Update the technical-context and Constitution Check entries to match the committed dependency cleanup.</comment>
<file context>
@@ -0,0 +1,152 @@
+
+**Language/Version**: Go 1.25.0 (pinned in `go.mod`)
+
+**Primary Dependencies**: no new dependency. Uses cobra, viper, logrus, and `pgx` (already used only to parse the connection URL). Deployment access stays through the `docker` and `kubectl` command-line tools (ADR 0001).
+
+**Storage**: none. The tool reads the task manager's PostgreSQL database (Prefect 3 schema) in read-only sessions and writes two JSON files into the bundle staging directory.
</file context>
| @@ -0,0 +1 @@ | |||
| `infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL. | |||
There was a problem hiding this comment.
P3: This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At changelog/181.changed.md, line 1:
<comment>This release note overstates partial-URL behavior: an omitted component can be filled from the task-manager container's Prefect URL before defaults are applied. Describe the URL-source precedence instead of saying every omitted part takes the default.</comment>
<file context>
@@ -0,0 +1 @@
+`infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL.
</file context>
| `infrahub-backup` now reads the task manager database user, password and database only from the Prefect connection URL itself. A part that the URL leaves out takes the default, instead of a value from `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user on the machine that runs the tool. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL. | |
| `infrahub-backup` now resolves task manager database credentials from the Prefect connection URL in the operator environment, then the task-manager environment, and finally the defaults; it never uses `PGUSER`, `PGPASSWORD`, `PGDATABASE`, `~/.pgpass` or the operating-system user for omitted URL parts. The warning about a URL that does not parse now names the part that does not parse, without quoting the URL. |
The pull request already wrote most of the spec into dev/: ADR 0008, the storage section of the collect knowledge page and the timeout-wrapping exception in the collector guidelines. This extraction adds what was still missing and archives the spec. ADR 0009 records that PostgreSQL is reached only through psql, pg_dump and pg_restore inside task-manager-db, over TCP to localhost, with the alternatives that research R1 rejected: a Go client, which has no network path through docker exec or kubectl exec; Unix-socket peer authentication, which the Bitnami image and CloudNativePG refuse for the owner; and the Prefect server container. The knowledge page gains the file contracts of storage.json and storage-breakdown.json, the manifest reason for each outcome, the five error kinds and the progress lines, and the privileges the queries need. The non-superuser prefect role from the Infrahub production deployment documentation was checked against postgres:18-alpine with password authentication required on localhost: both collectors and infrahub-backup create log in as that role. A gotcha records that the official image trusts localhost, so a wrong password still logs in on Docker Compose. The collector guidelines gain the one-JSON-value-per-query rule, the breakdown list as data, and how to test a database time limit end to end and a change to the password path. The spec moves to specs/archive/ with its extraction record, and the eight references to its path in ADR 0008, two Go comments and tests/helpers/bundle.py point at the archive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying infrahub-ops-cli with
|
| Latest commit: |
2380139
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://aa7de4ef.infrahub-ops-cli.pages.dev |
| Branch Preview URL: | https://fac-report-taskmgr-storage-u.infrahub-ops-cli.pages.dev |
There was a problem hiding this comment.
2 existing issues remain and 3 new issues found across 24 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="dev/knowledge/infrahub-collect.md">
<violation number="1" location="dev/knowledge/infrahub-collect.md:124">
P2: This outcome is incorrect for Docker deployments with a stopped-but-deployed `task-manager-db`: the collector is not skipped because `ServiceReplicas` includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.</violation>
</file>
<file name="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md">
<violation number="1" location="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md:9">
P3: This rationale is misleading: `docker exec` and `kubectl exec` run the command inside the target container, where `localhost:5432` reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.</violation>
<violation number="2" location="dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md:13">
P2: This decision overstates the implementation: `restorePostgreSQL` can use Unix-socket authentication and always targets the `postgres` maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.</violation>
</file>
Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
| | Case | `task-manager-db-storage` | `task-manager-db-storage-breakdown` | | ||
| |---|---|---| | ||
| | Neither the flag nor `INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE` is set | runs anyway | `skipped`, `not requested` | | ||
| | `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same | |
There was a problem hiding this comment.
P2: This outcome is incorrect for Docker deployments with a stopped-but-deployed task-manager-db: the collector is not skipped because ServiceReplicas includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/knowledge/infrahub-collect.md, line 124:
<comment>This outcome is incorrect for Docker deployments with a stopped-but-deployed `task-manager-db`: the collector is not skipped because `ServiceReplicas` includes stopped containers, and the subsequent exec fails instead. Document the no-container case, or explicitly distinguish stopped containers from an absent service.</comment>
<file context>
@@ -92,6 +92,59 @@ The size functions (`pg_total_relation_size`, `pg_table_size`, `pg_indexes_size`
+| Case | `task-manager-db-storage` | `task-manager-db-storage-breakdown` |
+|---|---|---|
+| Neither the flag nor `INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE` is set | runs anyway | `skipped`, `not requested` |
+| `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same |
+| The query, or the breakdown's catalog query, reaches its limit | `failed`, `timed out after 60s` | `failed`, `timed out after 60s`; no table is attempted |
+| `psql` is missing, the login is rejected, the server cannot be reached, or another error | `failed`, `psql missing: <cause>`, `login rejected: <cause>`, `not reachable: <cause>` or `query failed: <cause>` | the same, for the catalog query; no table is attempted |
</file context>
| | `task-manager-db` has no running replica | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same | | |
| | `task-manager-db` has no container | `skipped`, `service not deployed; the task manager may use an external PostgreSQL server` | the same | |
|
|
||
| ## Decision | ||
|
|
||
| Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container. |
There was a problem hiding this comment.
P2: This decision overstates the implementation: restorePostgreSQL can use Unix-socket authentication and always targets the postgres maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md, line 13:
<comment>This decision overstates the implementation: `restorePostgreSQL` can use Unix-socket authentication and always targets the `postgres` maintenance database, rather than TCP with the URL database and password. Limit this paragraph to collector queries or document the existing restore branches separately.</comment>
<file context>
@@ -0,0 +1,31 @@
+
+## Decision
+
+Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container.
+
+Queries run through `psql -h localhost -U <owner> -d <database> -X -q -A -t -v ON_ERROR_STOP=1 -c <query>`. `-X` ignores a `.psqlrc` file in the container, and `-A -t` prints the bare value without alignment or headers. Each query returns one JSON value that PostgreSQL builds with `json_build_object` and `json_agg`, and the tool parses that JSON. It never splits `psql` text output on delimiters, because flow names and event names can contain any character.
</file context>
| Every PostgreSQL command runs inside the `task-manager-db` container, which on Kubernetes is the pod that the shared pod lookup selects, the primary of a CloudNativePG cluster. It connects over TCP to `localhost` as the database owner, with the user, password and database that the Prefect connection URL holds. The host and port of that URL are ignored: the connection is always `localhost:5432` inside the container. | |
| The `infrahub-collect` storage queries run inside the `task-manager-db` container and connect over TCP to `localhost` as the database owner, using credentials resolved from the Prefect connection URL or documented fallbacks. The URL's host and port are ignored. `infrahub-backup` also runs `pg_dump` and `pg_restore` there, but its restore path may use Unix-socket authentication and the `postgres` maintenance database. |
|
|
||
| ## Context | ||
|
|
||
| The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL. |
There was a problem hiding this comment.
P3: This rationale is misleading: docker exec and kubectl exec run the command inside the target container, where localhost:5432 reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev/adr/0009-postgresql-access-through-psql-in-task-manager-db.md, line 9:
<comment>This rationale is misleading: `docker exec` and `kubectl exec` run the command inside the target container, where `localhost:5432` reaches PostgreSQL. Say that the CLIs do not expose the database port to the operator host, rather than saying they open no network path.</comment>
<file context>
@@ -0,0 +1,31 @@
+
+## Context
+
+The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL.
+
+## Decision
</file context>
| The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001), and `docker exec` and `kubectl exec` open no network path to the PostgreSQL port of `task-manager-db`. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #193, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL. | |
| The three binaries run on an operator's host, outside the deployment's network. They reach the deployment only through the `docker` and `kubectl` command-line tools (ADR 0001); those tools execute commands inside `task-manager-db`, where `localhost:5432` reaches PostgreSQL, but they do not expose that port directly to the host. `infrahub-backup` has always run `pg_dump` and `pg_restore` inside that container. The storage figures of the task manager database (spec 006) added read queries and had to choose how to run them. Until pull request #181, `github.com/jackc/pgx/v5` was a dependency, used only to parse the Prefect connection URL. |
Why
A bundle from
infrahub-collect createdoes not show how much disk space the task manager's PostgreSQL database (task-manager-db) uses, or which records use it. Support then has to send the customer SQL queries to run (#181).With this change, support can answer these questions from the bundle alone (PRD, spec):
Every bundle answers questions 1 to 3. Question 4 needs the new
--include-task-manager-storageflag, because answering it reads every row of those tables.Out of scope, as listed in the spec:
Closes #181
What changed
Storage figures in every bundle
task-manager-db-storage, writestask-manager-db/storage.json:Storage breakdown with
--include-task-manager-storageINFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.task-manager-db/storage-breakdown.json. It covers seven Prefect tables:events,flow_run,task_run,flow_run_state,task_run_state,logandartifact.failed, and the other tables are still collected. A table that the deployed Prefect version does not have is recorded asabsent.Failures
timed out after 60s,login rejected: ...,psql missing: ...ornot reachable: ....storage.err.txtorstorage-breakdown.err.txt, with the password masked.skippedwhen the deployment has notask-manager-dbservice.Change to
infrahub-backupinfrahub-backupreads the user, password and database only from the Prefect connection URL itself.PGUSER,PGPASSWORD,PGDATABASE,~/.pgpassor the operating-system user of the machine that runs the tool.Implementation notes
runTaskManagerDBQuery, which runspsqlinsidetask-manager-db(ADR 0008):statement_timeout), so a query stops on the server even after the tool stops waiting for it.psqlon stdin, through a shortshscript, so it is in no argument of thedocker,kubectl,shorpsqlprocess.task-managercontainer;POSTGRES_USERorPOSTGRES_PASSWORDintask-manager-db;url.Parseinstead ofpgx.ParseConfig, sogithub.com/jackc/pgx/v5is no longer a dependency. The NixvendorHashis updated.What stayed the same
infrahub-backup createare unchanged.infrahub-backup createstill passes the database password topg_dumpas adockerorkubectlargument.infrahub-collectflags are unchanged.Extraction of the spec into
dev/The last commit,
2380139, runsspeckit-opsmill-extracton the spec, so the PR lands with the knowledge extracted. The PR had already written ADR 0008, the storage section of the knowledge page and the timeout-wrapping exception of the guidelines; the commit adds what was still missing:psql,pg_dumpandpg_restoreinsidetask-manager-db, over TCP tolocalhost, with the alternatives that research R1 rejected (a Go client, Unix-socket peer authentication, the Prefect server container).dev/knowledge/infrahub-collect.md: the file contracts ofstorage.jsonandstorage-breakdown.json, the manifest reason for each outcome, the five error kinds and the progress lines, the privileges the queries need, and a gotcha: the officialpostgresimage trustslocalhost, so a wrong password still logs in on Docker Compose.dev/guidelines/collectors.md: one JSON value per query, the breakdown list as data, and how to test a database time limit and a change to the password path end to end.specs/archive/006-taskmgr-db-storage/with its extraction record. The references to it in ADR 0008, two Go comments andtests/helpers/bundle.pypoint at the archive.Suggested review order
src/internal/app/collect_taskmanager_db.go: credential lookup, query runner, error classification and masking.src/internal/app/collect_storage.go: the SQL and the two collectors.src/internal/app/app_config.go: the Prefect URL parsing, whichinfrahub-backupalso uses.ExecSeparateInputContextinenvironment_docker.goandenvironment_kubernetes.go.tests/e2e/test_docker_collect.pyandtests/e2e/test_k8s_collect.py.The files under
specs/archive/006-taskmgr-db-storage/are planning records. They can be skimmed. The last commit extracted the spec intodev/and archived it (see "Extraction of the spec intodev/" above).How to review
These parts need the most attention:
shscript that reads the password from stdin. On Kubernetes it depends onkubectl exec -i. This path has unit tests only: no end-to-end run has covered it yet (see "How to test").ACCESS SHARElock on the table it reads and on the tables it joins, for up to 5 minutes. During a task manager upgrade, a schema change on one of these tables waits until the query ends. The user documentation warns about this.These review findings are not addressed in this PR:
infrahub-collectcan log in totask-manager-dbwhileinfrahub-backup, and--include-backup, fall back to the defaults.localhost:5432insidetask-manager-db.event_resourcestable.The implementation report lists the other deferred findings.
How to test
Results:
make test,make vetand golangci-lint (0 issues) pass on419cd9f, and so dorumdl, Vale andnix build.6f21171, before the last two commits. The last two commits change how the password reachespsql, and CI runs both end-to-end suites on this PR.postgres:18-alpinecontainer that requires a password on localhost:@and$logs in.infrahub-collect.login rejected.t046-cnpg-check.md. It ran before the rebase onmainand was not run again.infrahub-backup createwere run against the least-privilege PostgreSQL setup of the Infrahub production deployment documentation ("Least-privilege service credentials"): a non-superuserprefectrole that owns theprefectdatabase and thepublicschema, onpostgres:18-alpinestarted with--auth-host=scram-sha-256, with the Infrahub 1.12.0 task manager connected as that role and a percent-encoded password holding@, a space,/,$and#. Both storage collectors weresuccess, PostgreSQL's connection log showed every session asuser=prefect database=prefect application_name=infrahub-collect,pg_dumpconnected asprefectas well, and a wrong password in the operator's environment was reported aslogin rejected.2380139changes only Markdown, two Go comments and comments intests/helpers/bundle.py.go vet, the unit tests,rumdlandruffpass on it.Impact & rollout
task-manager-db/storage.json, and the manifest lists two new collectors.infrahub-backupno longer takes missing parts of the Prefect URL fromPG*variables or~/.pgpass.--include-task-manager-storage, with the variableINFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUTlowers the query limits for the end-to-end tests only, and is not documented for operators.task-manager-dbneedsshandpsql, which the officialpostgres, Bitnami and CloudNativePG images include. Without them, the manifest records the collector as failed, and the bundle is still produced.Checklist
changelog/181.added.md,changelog/181.changed.md)docs/docs/collect/,docs/docs/reference/)dev/adr/0008-*,dev/adr/0009-*,dev/guidelines/collectors.md,dev/knowledge/infrahub-collect.md,AGENTS.md)🤖 Generated with Claude Code
Summary by cubic
Troubleshooting bundles now include the storage use of the task manager database, so support can answer how much disk the database uses and which records fill it without sending the customer SQL queries (closes #181).
Task manager database storage collectors
task-manager-db/storage.json: database and per-table sizes, live and dead row estimates, and last vacuum times.--include-task-manager-storage(orINFRAHUB_INCLUDE_TASK_MANAGER_STORAGE) addstask-manager-db/storage-breakdown.json: exact row counts and oldest/newest timestamps per record group for the flow run, task run, state, log, event and artifact tables.task-manager-dbwith time limits PostgreSQL enforces; a failed or missing table is recorded in the manifest without stopping the bundle, and the password reachespsqlon stdin, not in a process argument.infrahub-backupcredential lookupPG*variables,~/.pgpassor the OS user, andpgxis replaced withnet/urlparsing.Written for commit 419cd9f. Summary will update on new commits.