Skip to content

fix(logs): report a multiplexer outage once, with its cause - #351

Merged
alexei-led merged 1 commit into
mainfrom
fix/multiplexer-outage-logging
Oct 25, 2026
Merged

alexei-led merged 1 commit into
mainfrom
fix/multiplexer-outage-logging

Conversation

@alexei-led

Copy link
Copy Markdown
Owner

A few seconds of agterm unavailability logged a full traceback every second from the session-protocol poll, one skip line per second from the status poll, and one every two seconds from the session monitor. A longer outage buries the errors around it and never says which backend call failed.

  • ProtocolRuntime.poll: one warning when the outage starts, one info when it recovers, instead of a traceback per retry.
  • Status poll and session monitor: route the skip notice through log_throttled.
  • AgtermManager._tree: name the failing step (window list or tree --window <id>) and the window, so an unconfirmed listing has a diagnosable cause.

Tests: new test_runtime_outage_logging.py; assertions added to the agterm, polling coordinator, and session monitor suites. Full non-integration suite passes (8142 passed, 25 skipped); ruff, pyright, and the multiplexer boundary audits are clean.

A few seconds of agterm unavailability produced a full traceback every
second from the session-protocol poll, plus one skip line per second from
the status poll and one every two seconds from the session monitor. A
longer outage buries the errors around it under thousands of identical
lines and says nothing about which backend call failed.

- runtime.poll: log one warning when the outage starts and one info when
  it recovers, instead of a traceback per retry.
- status poll and monitor: route the skip notice through log_throttled.
- agterm _tree: name the failing step (window list or tree read) and the
  window, so an unconfirmed listing has a diagnosable cause.
@greptile-apps

greptile-apps Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Low impact] The change reaches recurring multiplexer polling logs in the bot.

The PR appears safe to merge, with a non-blocking issue in when recovery is reported.

Findings

  1. P2 Recovery reported too early ▶

Summary

This PR reduces repeated outage logs and identifies failed agterm listing calls.

  • Session protocol polling logs outage starts and recovery.
  • Status and session polling throttle unavailable-listing warnings.
  • agterm lookup warnings name the failed step and window.

Reviews (1) · Last reviewed commit: "fix(logs): report a multiplexer outage o..." · Reviewed by Greptile

Comment on lines +64 to +66
if unavailable:
unavailable = False
logger.info("Session protocol source recovered")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Recovery reported too early

poll reports recovery before queued hooks finish. During a transcript outage, refresh() can succeed because _update() catches the error, while queued permission hooks still fail in _match_hook(). With several such hooks queued, successive passes emit “recovered” followed by another warning and traceback for the same outage.

Move the recovery message and flag reset after the hook loop, and add a test with failing queued hooks.

@archfit-review

Copy link
Copy Markdown
Contributor

Archfit: Architecture needs attention · advisory · no baseline

Next: Advisories never block. Review the listed dimensions and diagnostics on the archfit/state check.

Details in Archfit

98 advisories · never block
Dependency This PR Where Strength Suggested move
api → session_protocol existing src/ccgram/api/server.py:18, src/ccgram/api/server.py:19, src/ccgram/api/server.py:20 and 3 more functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → api existing src/ccgram/bootstrap.py:420 functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → domain_config existing src/ccgram/bootstrap.py:30, src/ccgram/bootstrap.py:67, src/ccgram/cli.py:306 and 15 more functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → handlers existing src/ccgram/bootstrap.py:29, src/ccgram/bootstrap.py:32, src/ccgram/bootstrap.py:33 and 12 more intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
app_bootstrap → handlers existing src/ccgram/bootstrap.py:518 intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
app_bootstrap → hooks existing src/ccgram/hook.py:29, src/ccgram/hook.py:33, src/ccgram/hook.py:1155 and 2 more functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → multiplexer existing src/ccgram/bootstrap.py:548, src/ccgram/hook.py:37 functional Introduce a contract: lower the integration strength, so the target can change without breaking this module
app_bootstrap → multiplexer_backends existing src/ccgram/hook.py:35, src/ccgram/hook.py:36, src/ccgram/main.py:223 functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → providers existing src/ccgram/bootstrap.py:52, src/ccgram/doctor_cmd.py:25, src/ccgram/doctor_cmd.py:182 and 1 more functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → session_protocol existing src/ccgram/bootstrap.py:396, src/ccgram/cli.py:303, src/ccgram/cli.py:357 and 2 more functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → session_state existing src/ccgram/bootstrap.py:53, src/ccgram/bootstrap.py:56, src/ccgram/bootstrap.py:65 and 3 more intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
app_bootstrap → session_state existing src/ccgram/bootstrap.py:49, src/ccgram/bootstrap.py:399, src/ccgram/bootstrap.py:523 intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
app_bootstrap → telegram_adapter existing src/ccgram/doctor_cmd.py:26, src/ccgram/main.py:50, src/ccgram/main.py:314 functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → terminal_adapter existing src/ccgram/bootstrap.py:55, src/ccgram/bootstrap.py:359 functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → transcript existing src/ccgram/bootstrap.py:66 functional Move the functionality: lower the distance, so the coupled code lives in one module
app_bootstrap → window_state existing src/ccgram/bootstrap.py:529, src/ccgram/status_cmd.py:30 functional Move the functionality: lower the distance, so the coupled code lives in one module
domain_config → handlers existing src/ccgram/utils.py:418 functional Move the functionality: lower the distance, so the coupled code lives in one module
domain_config → multiplexer_backends existing src/ccgram/config.py:274 functional Move the functionality: lower the distance, so the coupled code lives in one module
domain_config → providers existing src/ccgram/command_catalog.py:11 functional Move the functionality: lower the distance, so the coupled code lives in one module
domain_config → session_protocol existing src/ccgram/utils.py:32 functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → domain_config existing src/ccgram/handlers/agent_command.py:31, src/ccgram/handlers/callback_registry.py:21, src/ccgram/handlers/callback_tokens.py:19 and 72 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → llm existing src/ccgram/handlers/hook_events.py:178, src/ccgram/handlers/shell/shell_capture.py:517, src/ccgram/handlers/shell/shell_commands.py:34 and 1 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → miniapp existing src/ccgram/handlers/status/status_bar_actions.py:32 functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → multiplexer existing src/ccgram/handlers/polling/window_tick/apply.py:30, src/ccgram/handlers/polling/window_tick/observe.py:21, src/ccgram/handlers/polling/window_tick/observe.py:24 functional Introduce a contract: lower the integration strength, so the target can change without breaking this module
handlers → providers existing src/ccgram/handlers/agent_command.py:215, src/ccgram/handlers/agent_command.py:281, src/ccgram/handlers/commands/forward.py:28 and 39 more intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
handlers → providers existing src/ccgram/handlers/recovery/resume_command.py:42 intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
handlers → session_protocol existing src/ccgram/handlers/commands/failure_probe.py:30, src/ccgram/handlers/interactive/interactive_callbacks.py:18, src/ccgram/handlers/recovery/recovery_banner.py:44 and 4 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → session_state existing src/ccgram/handlers/agent_command.py:32, src/ccgram/handlers/agent_command.py:199, src/ccgram/handlers/agent_command.py:321 and 50 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → session_state_ports existing src/ccgram/handlers/hook_events.py:25, src/ccgram/handlers/status/status_bubble.py:22 functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → telegram_adapter existing src/ccgram/handlers/commands/menu_sync.py:23, src/ccgram/handlers/commands/panel.py:23, src/ccgram/handlers/dashboard_command.py:12 and 41 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → terminal_adapter existing src/ccgram/handlers/agent_command.py:35, src/ccgram/handlers/callback_helpers.py:10, src/ccgram/handlers/callback_registry.py:22 and 47 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → transcript existing src/ccgram/handlers/last_reply.py:51, src/ccgram/handlers/polling/polling_state.py:163, src/ccgram/handlers/polling/polling_state.py:202 and 1 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → tts existing src/ccgram/handlers/messaging_pipeline/message_queue.py:40 functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → whisper existing src/ccgram/handlers/voice/voice_handler.py:21, src/ccgram/handlers/voice/voice_handler.py:22 functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → window_state existing src/ccgram/handlers/commands/__init__.py:25, src/ccgram/handlers/commands/failure_probe.py:28, src/ccgram/handlers/commands/forward.py:31 and 41 more functional Move the functionality: lower the distance, so the coupled code lives in one module
handlers → window_state_ports existing src/ccgram/handlers/agent_command.py:36, src/ccgram/handlers/cleanup.py:25, src/ccgram/handlers/hook_events.py:24 and 19 more functional Move the functionality: lower the distance, so the coupled code lives in one module
hooks → providers existing src/ccgram/hooks/adapters.py:15 functional Move the functionality: lower the distance, so the coupled code lives in one module
llm → domain_config existing src/ccgram/llm/__init__.py:62 functional Move the functionality: lower the distance, so the coupled code lives in one module
llm → session_protocol existing src/ccgram/llm/summarizer.py:14 functional Move the functionality: lower the distance, so the coupled code lives in one module
miniapp → domain_config existing src/ccgram/miniapp/api/terminal.py:32 functional Move the functionality: lower the distance, so the coupled code lives in one module
miniapp → session_state existing src/ccgram/miniapp/api/transcript.py:53 functional Move the functionality: lower the distance, so the coupled code lives in one module
miniapp → window_state_ports existing src/ccgram/miniapp/api/terminal.py:98 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer → multiplexer_backends existing src/ccgram/multiplexer/__init__.py:23 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer → session_state existing src/ccgram/multiplexer/vim_state.py:20 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer → window_state existing src/ccgram/multiplexer/reconciliation.py:11, src/ccgram/multiplexer/window_liveness.py:14 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer_backends → domain_config added src/ccgram/multiplexer/agterm.py:53, src/ccgram/multiplexer/tmux.py:38 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer_backends → multiplexer existing src/ccgram/multiplexer/herdr.py:72 intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
multiplexer_backends → multiplexer existing src/ccgram/multiplexer/tmux.py:51 intrusive Introduce a contract: lower the integration strength, so the target can change without breaking this module
multiplexer_backends → provider_detection existing src/ccgram/multiplexer/agterm.py:195, src/ccgram/multiplexer/agterm_panes.py:55 functional Move the functionality: lower the distance, so the coupled code lives in one module
multiplexer_backends → providers existing src/ccgram/multiplexer/agterm.py:520 functional Move the functionality: lower the distance, so the coupled code lives in one module

On every row: Imbalanced coupling · critical · across modules · high volatility.

  • and 48 more

Why: functional coupling across a module boundary to a volatile target: a change on one side forces a change on the other (Balanced Coupling: critical severity)

Why: intrusive (implementation-level) coupling across a module boundary to a volatile target: a change on one side forces a change on the other (Balanced Coupling: critical severity)

Evidence and inputs

Decision

  • Engine verdict needs attention: hard gates pass, 0 active blockers, every required rule evaluated
  • The gate is advisory for this repository: the engine's verdict does not block merging.
  • Attention only (architecture): dimensions not fully measured: intent, coupling, testability, operations, drift; active diagnostics: 98 · code attention_only

Dimensions

Dimension Status Measured Confidence Result
intent partial 6/7 ("declared rules evaluated") medium pass
structure measured 1281/1529 ("discovered dependencies resolved inside the declared module map") high warn
modularity measured 4/24 ("declared modules with a declared public surface") low pass
coupling partial 688/707 ("cross-boundary edges scored") medium warn
change_locality measured 24/24 ("declared modules touched in the scanned history window") high pass
complexity measured 24/24 ("declared modules with complete dependency-chain and degree values") high pass
testability partial 544/544 ("classified source files") medium pass
operations partial 0/24 ("declared modules with a corroborated deploy unit and qualifying owner") medium pass
drift unmeasured 0/0 unrated not applicable

Evidence gaps

  • Analyzers that did not run (absent or disabled): go/packages, dependency-cruiser, cargo, ast-grep, cargo-modules
  • Dimension intent, unknown fact: active rule conformance — one or more active rules lack the completed producer evidence their checks require: no-import-cycles
  • Dimension structure, unknown fact: external dependency structure — 248 dependencies leave the declared module map (248 on libraries, 0 to or from first-party code no module owns), so their direction and layer are outside this claim
  • Dimension coupling, unknown fact: coupling strength — integration strength is unknown on one or more cross-boundary candidates, so no rung is invented
  • Dimension coupling, unknown fact: coupling distance — architectural distance is unknown on one or more cross-boundary candidates, so no rung is invented
  • Dimension change_locality, unknown fact: essential vs accidental volatility — commit frequency corroborates a declared volatility; it cannot establish one
  • Dimension complexity, unknown fact: cognitive complexity — no cognitive-complexity analyzer is claimed; module-graph shape is the architecture-level measure
  • Dimension testability, unknown fact: supplied coverage units — coverage is disabled, so no supplied coverage units were observed
  • Dimension testability, unknown fact: coverage path resolution — coverage is disabled, so no supplied coverage paths were available to resolve
  • Dimension testability, unknown fact: coverage module attribution — coverage is disabled, so no supplied coverage was available to attribute to declared modules
  • Dimension testability, unknown fact: coverage freshness — coverage is disabled, so no supplied coverage freshness could be established
  • Dimension operations, unknown fact: corroborated deploy unit — one or more declared modules have no independently corroborating deploy manifest
  • Dimension operations, unknown fact: observed runtime topology — committed manifests corroborate declared deploy units; they do not observe what is actually running
  • Dimension operations, unknown fact: supply-chain inventory — SBOM and vulnerability facts are a separate report family and have no collector in v1
  • Dimension drift, unknown fact: admissible persisted reference — no comparable architecture-state reference is stored
  • Dimension drift, unknown fact: complete two-sided seam identity — two-sided seam identity cannot be compared without an admissible persisted reference
  • No baseline: every finding counts as new, including older debt. New does not mean added by this pull request.

Inputs and identity

  • Policy digest: 79a977c51293abbe230e69f6ba5984d37dcdf60833336a7f5557d6f54a1d1b6f
  • Baseline digest: no baseline
  • Labels digest: no labels file
  • Measurement profile: version archfit.measurement.v2, settings hash 0bb81030aa53141846de3953d5df392c8ab250618e04f4388b06d4dc3016b0cb
  • Producer identity, claimed: engine v3.2.0, image sha256:5f2d4a5fff1ace10d7b08937429179631ae0c1d0e1f9e6f37c72b30954cf00db, linux/amd64

Measurement is customer-attested: the producer is authenticated, the report content is not independently checked.

For coding agents

Agent tasks

[]

@alexei-led
alexei-led merged commit bfe7bb7 into main Oct 25, 2026
5 checks passed
@alexei-led
alexei-led deleted the fix/multiplexer-outage-logging branch October 25, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant