Customer-facing connection health for messaging channels (push first)

[dmarchuk]

Problem

When a customer's messaging channel breaks because of a credential or config problem (an expired APNs signing key, a bad or rotated Firebase service account, a wrong project or topic), sends fail and the customer gets no signal. Today the only path is the team noticing an internal alert, looping in support, and support relaying to the customer. We act as a human proxy for a problem only the customer can fix.

Raised in review on the push alert runbooks: https://github.com/PostHog/charts/pull/14699#discussion_r3892203126. The push credential-failure runbook currently escalates to the team, which is a beta stopgap, not the durable answer.

Proposal

Surface channel/connection health to the customer so they self-serve, and reuse the surface we already have for email instead of building a push-only one-off.

We already have the shape of this on the email side:

• Bounce-driven suppression health (message_suppression): auto-suppress an address after N consecutive bounces, keep the SMTP diagnostic, reset on a successful delivery.
• Deliverability surfaces in the messaging UI (Reputation, Suppression list, Opt-outs tabs).

Push should be another signal into the same channel-health surface, not a parallel system.

Phase 1 (push, MVP)

• A health state on the push integration/channel (healthy or unhealthy, with reason, since, and last error), set from the native send-failure path. The reason is already classified there (auth_error, invalid_token, ...), and cleared on the next successful send.
• A badge and banner on Messaging › Channels: "This channel isn't sending: APNs key expired. Reconnect." with a reconnect action.

Phase 2 (proactive and general)

• Proactive in-app or email notification to project admins when a channel goes unhealthy, reusing the existing notification mechanism.
• Generalize the health-state + reason + fix-action model so OAuth CDP destinations (token expiry/revocation), warehouse sources (sync auth failures), and email all plug into one surface. Build on the existing signals rather than a new parallel system: the CDP hog watcher already degrades and disables failing destinations, and warehouse sources already show sync status.

Key design nuance

Notify only on channel-level breakage.

• auth_error is always actionable: the channel's credentials are rejected.
• A invalid_token that is per-device churn (an uninstalled app) is normal and already auto-pruned (push_notification_token_pruned_total). Do not notify on it, or customers get spammed.
• Only a sustained or widespread invalid_token (the whole channel pointing at the wrong project or topic) is a config problem worth surfacing.

Debounce so the state does not flap: mark unhealthy after sustained failures, and healthy again on the next success.

Scope notes

• Build for the native push send path, where the reason is classified, not the webhook/HTTP-request path.
• Design the health state to be integration-agnostic from the start, and ship push as the first vertical.

Out of scope

The invalid_api_key push-subscription rejections are a different thing: apps that ship a token resolving to no project. Those senders never configured a channel, so this feature does not apply to them.