Repository navigation
Conversation
…s leave across restarts On Linux with a systemd user manager, each command of bash, screen and a routine runs in a transient scope of its own (systemd-run --user --scope), in one slice per Pots home (app-pots-<key>.slice). When memory runs out, systemd-oomd or the kernel stops the command that uses it, not the gateway with every bot in it. The command stays the gateway's child, in its own process group, with the same pipes, so output, timeouts and interrupts work as before; where systemd-run cannot start a scope, the command runs as before, never twice. Without a user manager (macOS, Windows, system services, containers), nothing changes. Restarts: a command Pots still waits on when a bot stops is stopped with its scope, also at the next start after a crash (its own process still runs in its scope); what a command left running when it returned keeps running, and the bot's next wake lists it. When the gateway's own user service is stopped on purpose (a stop job, not a restart), the slice stops too. The prompt states these facts and the scopes' names. The service unit gets OOMPolicy=continue, and pots service restart adds it to a unit an older version wrote.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec7502ff3a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const stopped = scopes.filter((scope) => commandRuns(scope)); | ||
| // SIGTERM, and SIGKILL for what is left after the scope's stop timeout. One that ended meanwhile is not loaded any | ||
| // more: systemctl says so and stops the others. | ||
| if (stopped.length > 0) await systemctl(["stop", "--no-block", "--", ...stopped.map((scope) => scope.unit)]).catch(() => {}); |
There was a problem hiding this comment.
After a gateway crash and automatic service restart, an in-flight command that ignores SIGTERM or performs lengthy shutdown work can continue running after recover() returns. systemctl --help describes --no-block as “Do not wait until operation finished”, so this await only queues the stop; Bot.#open then proceeds to initialize the gateway and agent while the old command remains alive until termination or systemd's stop timeout. This allows old writers or resource holders to interfere with newly resumed work. Wait for the recovered scopes to become empty, or explicitly terminate them and verify completion, before allowing the bot to start commands.
Useful? React with 👍 / 👎.
| async stopInterrupted(): Promise<number> { | ||
| const tokens = [...this.#interrupted]; | ||
| this.#interrupted.clear(); | ||
| if (tokens.length > 0) await systemctl(["stop", "--no-block", "--", ...tokens.map((token) => `${this.prefix}-*-${token}.scope`)]).catch(() => {}); |
There was a problem hiding this comment.
With bots named a and a-1, stopping bot a can also terminate bot a-1's commands when their tokens coincide. Tokens use a timestamp and a per-bot counter, so commands started by different bots in the same millisecond can share one. The stop pattern pots-<key>-a-*-<token>.scope then matches both bots because its * also consumes 1-<pid>. The systemctl documentation confirms that “normal shell-style globbing rules are used”. Unlike #list(), this cleanup does not enforce the exact bot boundary, so removing one bot or restarting the gateway can kill another bot's background processes that should survive. Resolve and filter exact unit names before stopping them, or retain each interrupted command's full unit name.
Useful? React with 👍 / 👎.
| "exec 3>&2 2>/dev/null", | ||
| "trap 's=1' USR1", | ||
| "trap : TERM", | ||
| `(exec ${quote(systemdRun)} --user --scope --quiet --collect --slice=${quote(slice)} --unit="${this.prefix}-$BASHPID-$2" --description=${quote(`Command of Pots bot ${bot}`)} -- ${quote(bash)} -c ${quote(inner)} pots "$1")`, |
There was a problem hiding this comment.
On systemd 258 and 259, this invocation expands environment references in the command argument before bash interprets it. The systemd 258 documentation describes expansion as “enabled (the default)” and specifies that scope expansion is performed by systemd-run itself. For example, value=ok; echo "${value}" becomes value=ok; echo "" when value is absent from the inherited environment; quoted script contents containing such references are also modified. This silently changes commands from bash, screen and routines on these systems, while the availability probe still succeeds. Explicitly pass --expand-environment=no so bash receives the original command unchanged.
Useful? React with 👍 / 👎.
On Linux with a systemd user manager, each command the agent runs (bash, screen, codemode's bash, and a routine's
--run) now runs in its own transient systemd scope. When memory runs out, oomd or the kernel then stops the heavy command instead of the whole gateway. A gateway restart keeps what commands left running. This implements Q1 (A) and Q2 (丙) from the 2026-11-13 OOM decision page. Without a user manager (macOS, Windows,--systemservices, containers), commands run as before.How a command runs (
src/agent/scopes.ts)ScopedExecutionEnvextends pi-durable'sNodeExecutionEnvand changes onlyexecfor shell strings. pi-durable is unchanged. The harness env (bash, screen, codemode, children, memory forks) and the routine runner both use it.(exec systemd-run --user --scope --quiet --collect --slice=app-pots-<key>.slice --unit=pots-<key>-<bot>-$BASHPID-<token> -- bash -c …). systemd-runexecs in place, so the command keeps the same PID, process group and pipes. Output, timeouts, interrupt andkill(-pgid)work as before.exit 143.<key>is the first 8 hex digits of sha256(POTS_HOME), so two homes never share scopes. Measured cost: about 25 ms per command, plus 1–2 user-journal lines. The scope description is generic (Command of Pots bot <id>), so command text, which can contain secrets, stays out of the journal.Restart lifecycle (Q2 丙)
setsidchildren. A command that ends aborted while the bot is closing counts as one Pots was waiting on.stopjob, not arestartjob, and whether it is that service's MainPID. If so, it stops the home's slice after the bots stop. This coverssystemctl --user stop pots,pots service uninstall(disable --now) and logout/shutdown.systemctl restart,pots service restart,pots service installand Restart=always keep the slice. Tested on systemd 255:PartOf=andStopPropagatedFrom=both also fire on restart and crash-restart, so neither fits.systemctl --user list-units/status/stopcommands. It states facts only, no rules.Unit
OOMPolicy=continueis added to the generated unit, both user and system.pots service restart, which is the documented upgrade step, adds the line to a unit that Pots wrote and that lacks it, then runs daemon-reload. It never touches a unit Pots did not write.Not done
stopLeftoverTunnel. That does not fit this change, so it is left as a follow-up.Tests
test/scopes.test.ts(runs only wheresystemd-run --user --scopeworks), with real scopes:systemctl restartkeeps the left process and stops the in-flight one;systemctl stopends everything.XDG_RUNTIME_DIR: commands run as before, with no prompt line and no note.test/service.test.ts: the unit hasOOMPolicy=continue; restart upgrades an older Pots unit once and leaves a foreign unit alone.TestGateway: anenvoption, anddispose()stops the home's slice.On this host:
npm run checkpasses.npm testpassed 315, skipped 2, failed 0 (rebased onto 0.7.2 afterwards; that rebase changed only CHANGELOG and package.json). CI passes on all six jobs. GitHub's ubuntu runner has a user manager, soscopes.test.tsran there too, including the transient-service test. The fallback is covered by the test withoutXDG_RUNTIME_DIRand by macOS and Windows, where the scope code never runs.