Skip to content

data-model.md

Latest commit

 

History

History
118 lines (102 loc) · 4.98 KB

data-model.md

File metadata and controls

118 lines (102 loc) · 4.98 KB
Raw

HookRelay — MongoDB data model

Three collections. Design goals: the live inspector's hot paths (recent events list, event detail, DLQ list) must be single-query and index-covered; delivery state must be updatable without cross-document transactions.

endpoints — configuration

{
  "_id": ObjectId,
  "slug": "demo-stripe",            // URL path segment: /ingest/{slug}
  "name": "Stripe (demo)",
  "description": "…",
  "provider": "STRIPE",             // STRIPE | GITHUB | GENERIC
  "secret": "whsec_…",              // HMAC secret; never returned by the API
  "signatureHeader": null,          // GENERIC only: custom header override
  "rejectInvalidSignature": false,  // kept for future strict mode; rejection is default when secret set
  "forwardUnverified": false,       // escape hatch: forward even if signature fails
  "maxAttempts": 3,                 // null → global default
  "active": true,
  "destinations": [
    {
      "id": "665f…",                // stable id, referenced by deliveries
      "name": "billing-service",
      "url": "https://internal.example/hooks/billing",
      "active": true,
      "transformCode": "return {…}",// null → payload forwarded verbatim
      "headers": { "X-Api-Key": "…" } // extra headers added on forward
    }
  ],
  "createdAt": ISODate, "updatedAt": ISODate
}

Indexes: {slug: 1} unique — resolved on every ingest.

events — one document per received webhook (immutable capture + aggregate status)

{
  "_id": ObjectId,
  "endpointId": "…", "endpointSlug": "demo-stripe", "provider": "STRIPE",
  "dedupeKey": "evt_1Q…",           // provider event id; absent if provider gives none
  "headers": { "stripe-signature": "t=…,v1=…", … },  // lowercased keys
  "contentType": "application/json",
  "body": "{ raw request body, capped at 1 MB }",
  "bodyBytes": 2481,
  "sourceIp": "203.0.113.7",
  "signatureStatus": "VALID",       // VALID | INVALID | NO_SIGNATURE | NO_SECRET
  "signatureDetail": null,          // human-readable reason when not VALID
  "status": "DELIVERED",            // PENDING | DELIVERED | PARTIAL | FAILED | REJECTED | RECEIVED
  "deliveryStats": { "total": 2, "delivered": 2, "dead": 0 },  // denormalized for the list view
  "receivedAt": ISODate
}

Indexes:

Index Serves
{receivedAt: -1} live feed default view (+ TTL variant for retention)
{endpointId: 1, receivedAt: -1} feed filtered by endpoint
{status: 1, receivedAt: -1} feed filtered by status
{endpointId: 1, dedupeKey: 1} unique, partial (dedupeKey $exists) race-safe ingestion dedupe (ADR-006). Partial because most GENERIC events carry no key — a sparse/compound combo would still index nulls and break uniqueness.

deliveryStats is denormalized so the list view never joins deliveries; it is recomputed transactionally-enough (single doc update) each time a delivery reaches a terminal state.

deliveries — one document per (event × destination); attempts embedded

{
  "_id": ObjectId,
  "eventId": "…",
  "endpointId": "…", "endpointSlug": "demo-stripe", "provider": "STRIPE", // denormalized: DLQ view needs no join
  "destinationId": "665f…", "destinationName": "billing-service",
  "url": "https://internal.example/hooks/billing",   // snapshot at fan-out time
  "status": "RETRYING",             // PENDING | RETRYING | DELIVERED | DEAD
  "hasTransform": true,
  "transform": { "status": "applied", "error": null, "durationMs": 3 },
  "attemptCount": 2,
  "maxAttempts": 6,
  "attempts": [                     // embedded: bounded by maxAttempts (≤ ~10), small
    { "number": 1, "at": ISODate, "responseStatus": 500,
      "responseBody": "…≤2KB…", "error": null, "durationMs": 187, "manual": false },
    { "number": 2, "at": ISODate, "responseStatus": null,
      "responseBody": null, "error": "HttpTimeoutException: …", "durationMs": 10004, "manual": false }
  ],
  "nextRetryAt": ISODate,           // inspector shows the countdown
  "lastAttemptAt": ISODate,
  "deadReason": null,
  "eventReceivedAt": ISODate,       // for TTL retention aligned with events
  "createdAt": ISODate
}

Indexes:

Index Serves
{eventId: 1} event detail page (all deliveries of an event)
{status: 1, lastAttemptAt: -1} DLQ view (status: DEAD), retrying view
{endpointSlug: 1, status: 1} per-endpoint health queries

Why attempts are embedded, not a collection: attempts per delivery are hard-capped by maxAttempts and each is ≤ ~2.3 KB (response snippet is truncated), so the document stays far from the 16 MB limit; the inspector always reads attempts with their delivery; and appending is a single $push-equivalent update. A separate collection would buy nothing and cost a join on the hottest detail view.

Retention

HOOKRELAY_RETENTION_DAYS > 0 creates TTL indexes on events.receivedAt and deliveries.eventReceivedAt. Default 0 (keep forever) — a debugging tool should not silently discard evidence.