# MailHub Delivery Webhooks Design

## Goal

Let each user push **terminal email delivery results** (`sent`, `bounced`, `failed`) to external business systems via signed HTTPS callbacks, with an observable SQLite-backed delivery queue (retry, logs, manual replay). Account-level endpoints are the default; domains may define override endpoints.

## Product Decisions

| Item | Choice |
|------|--------|
| Primary use case | Notify business systems of delivery outcomes |
| Events (MVP) | Terminal only: `sent`, `bounced`, `failed` |
| Scope | Account default + optional domain override |
| Reliability | In-process worker + SQLite queue, exponential backoff, delivery log, manual replay |
| Endpoints | Multiple URLs per account and per domain; each subscribes to a subset of events |
| Stack fit | Node app + SQLite; no Redis / external broker |

## Non-Goals (MVP)

- Intermediate events (`queued`, `deferred`)
- Per-recipient split payloads beyond fields already on `send_events`
- Workflow builders / conditional routing UI
- Guaranteed exactly-once delivery (at-least-once with idempotent delivery keys)
- Redis / BullMQ / separate worker service
- Webhook for DNS health or admin audit events

## Architecture

```text
send_event reaches terminal status
        │
        ▼
 enqueueWebhookDeliveries(sendEvent)
   resolve targets (domain override vs account)
   insert webhook_deliveries (pending), idempotent
        │
        ▼
 webhook worker (interval inside app process)
   claim due rows → HTTP POST signed body
   success | schedule retry | mark dead
```

### Trigger points

Concrete call sites in the current codebase (plan should wire both):

1. **`updateSendEventDelivery`** (Postfix tracker path) — when computed `nextStatus` is terminal **and** differs from the previous row status.
2. **`logSendEvent`** (and any wrapper that inserts a terminal status, typically `failed` on immediate send failure) — when the inserted status is already terminal.

Other mailer/submission/API paths only matter if they write through these functions; do not add parallel enqueue call sites without going through DB helpers.

Enqueue is **synchronous DB insert only** (must not await outbound HTTP). Worker performs HTTP.

### Process model

- Worker starts with HTTP server lifecycle (e.g. after listen).
- Poll interval: ~5–15s (constant).
- Concurrency: small fixed pool (e.g. 2–3 in-flight POSTs) to protect local Node and remote endpoints.
- On process restart, pending/retryable rows remain in SQLite and resume via `next_attempt_at`.

### Delivery status machine

`webhook_deliveries.status` values:

| Status | Meaning |
|--------|---------|
| `pending` | Not yet successfully delivered; may be due when `next_attempt_at <= now` |
| `processing` | Claimed by worker for an in-flight HTTP attempt (short-lived lease) |
| `success` | Last attempt received HTTP 2xx |
| `dead` | Exhausted max attempts without 2xx |

There is **no** long-lived `failed` status. Transient HTTP/network failures stay **`pending`** with an updated `next_attempt_at`, `attempt_count`, `error`, and `response_*` fields for the UI (“last attempt failed” is derived from `error` / `response_status` while `status=pending` or `dead`).

Transitions:

1. Enqueue → `pending`, `attempt_count=0`, `next_attempt_at=now`.
2. Worker **claims** due `pending` rows → `processing` (see claim protocol).
3. Complete HTTP attempt: always increment `attempt_count` (including 2xx).
4. HTTP 2xx → `success`.
5. HTTP non-2xx / timeout / network error:
   - If `attempt_count < 8` → `pending` + backoff `next_attempt_at`.
   - Else → `dead`.
6. Manual replay (from `success` or `dead`, or `pending` with errors) → `pending`, `attempt_count=0`, `next_attempt_at=now`, clear `error` (MVP may clear response preview). **Reject or no-op if status is `processing`** (wait for lease expiry) to avoid racing an in-flight POST.

### Worker claim protocol

To avoid double POST within one process (and reduce multi-instance races):

1. In a single SQLite transaction, `SELECT` up to N rows where `status='pending' AND next_attempt_at <= now` ordered by `next_attempt_at`, then `UPDATE` those ids to `status='processing'`, set `last_attempt_at=now`, and set `next_attempt_at` to a **lease deadline** (e.g. now + 2 minutes) so a crashed worker does not leave rows stuck forever.
2. Only claimed rows may perform HTTP POST.
3. On completion, update to `success` or back to `pending`/`dead` as above; clear lease by writing the final status.
4. Reaper (same worker loop): if `status='processing'` and `next_attempt_at < now` (lease expired), reset to `pending` with `next_attempt_at=now` so the row can be retried.

MVP assumes a **single app instance** (current Docker deploy). Claim still required for in-process concurrency.

## Resolution Rules

Given `userId`, `domainId`, `eventType` ∈ {sent, bounced, failed}:

1. Load **enabled** webhooks for this user where `domain_id = domainId` and `events` includes `eventType`.
2. If any such domain-scoped webhooks exist → **use only those** (override account).
3. Else load **enabled** account-scoped webhooks (`domain_id IS NULL`) that subscribe to `eventType`.
4. Skip disabled or unsubscribed endpoints.
5. For each selected webhook, insert a delivery row if not already present for  
   `(webhook_id, send_event_id, event_type)`.

## Data Model

### `webhooks`

| Column | Type | Notes |
|--------|------|--------|
| id | INTEGER PK | |
| user_id | INTEGER NOT NULL | FK users |
| domain_id | INTEGER NULL | NULL = account-level; else domain override |
| name | TEXT NOT NULL | Display label |
| url | TEXT NOT NULL | HTTPS endpoint |
| secret_ciphertext | TEXT NOT NULL | Encrypted, same pattern as DNS/SMTP secrets |
| secret_prefix | TEXT NOT NULL | For UI display |
| events_json | TEXT NOT NULL | JSON array subset of sent/bounced/failed |
| enabled | TEXT/INTEGER | boolean |
| created_at / updated_at | TEXT | ISO |

Indexes: `(user_id)`, `(user_id, domain_id)`.

### `webhook_deliveries`

| Column | Type | Notes |
|--------|------|--------|
| id | INTEGER PK | Public id may be exposed as `whd_{id}` |
| webhook_id | INTEGER NOT NULL | |
| user_id | INTEGER NOT NULL | Denormalized for isolation queries |
| send_event_id | INTEGER NOT NULL | Real event id, or `0` for synthetic tests; **no FK** to send_events |
| event_type | TEXT NOT NULL | sent \| bounced \| failed |
| payload_json | TEXT NOT NULL | Exact body bytes basis (JSON text) |
| status | TEXT NOT NULL | pending \| processing \| success \| dead |
| attempt_count | INTEGER NOT NULL DEFAULT 0 | |
| next_attempt_at | TEXT NOT NULL | |
| last_attempt_at | TEXT | |
| response_status | INTEGER | |
| response_body_preview | TEXT | Truncated (~1–2KB) |
| error | TEXT | |
| created_at | TEXT | |

Constraints / indexes:

- **UNIQUE** `(webhook_id, send_event_id, event_type)` for enqueue idempotency
- `(status, next_attempt_at)` for worker
- `(user_id, created_at DESC)` for UI logs

On webhook delete: CASCADE or mark deliveries orphaned—prefer **ON DELETE CASCADE** for simplicity.

## Retry Policy

| Attempt after first failure | Delay (approx.) |
|-----------------------------|-----------------|
| 1 | 1 minute |
| 2 | 5 minutes |
| 3 | 30 minutes |
| 4 | 2 hours |
| 5 | 6 hours |
| 6+ | 12 hours |

- Max attempts: **8** (including first try). Then `status = dead`.
- Success: HTTP **2xx**. All non-2xx and transport errors use the same backoff path (no special-case 4xx in MVP).
- Manual **replay**: mutate the **same** delivery row — `status=pending`, `attempt_count=0`, `next_attempt_at=now`, clear `error` / response preview; **do not change `payload_json`** so the signed body stays stable for that delivery id.

## HTTP Contract

### Request

- Method: `POST`
- Headers:
  - `Content-Type: application/json`
  - `User-Agent: MailHub-Webhook/1.0`
  - `X-MailHub-Signature: t=<unix_seconds>,v1=<hex_hmac>`
  - `X-MailHub-Event: email.sent|email.bounced|email.failed`
  - `X-MailHub-Delivery: whd_<id>`
- Timeout: 10 seconds
- Redirects: do not follow

### Body

```json
{
  "id": "whd_42",
  "type": "email.sent",
  "created_at": "2026-07-09T12:00:00.000Z",
  "data": {
    "message_id": "mh-42",
    "send_event_id": 42,
    "queue_id": "A1B2C3D4E5",
    "status": "sent",
    "domain": "example.com",
    "from": "noreply@example.com",
    "to": ["user@example.com"],
    "subject": "Hello",
    "detail": "optional status detail",
    "delivered_at": "2026-07-09T12:00:01.000Z"
  }
}
```

| Internal status | `type` |
|-----------------|--------|
| sent | `email.sent` |
| bounced | `email.bounced` |
| failed | `email.failed` |

### Signature

**Payload id construction:** In **one SQLite transaction**, insert the delivery row (placeholder `payload_json` if needed), read `lastInsertRowid`, then set final `payload_json` with `"id": "whd_<id>"` and stable `created_at` **before commit**, so the worker never claims a row without the signed body. Signature always uses that stored `payload_json` as `raw_body`.

Signed string: `{t}.{raw_body}` where `raw_body` is the exact stored JSON string POSTed.

```text
v1 = hex(HMAC_SHA256(secret, signed_string))
```

Receivers should reject if `|now - t| > 300` seconds (document 5-minute skew).

Secret: generated on create / rotate; returned **once** in API response; stored encrypted; list UI shows prefix only.

### SSRF protections

- Default: only `https:` URLs.
- Optional env flag (e.g. `WEBHOOK_ALLOW_HTTP_LOCAL=1`) for `http://127.0.0.1` / localhost in development.
- After DNS resolve, block private, loopback, link-local, and metadata IPs (document exceptions for local flag).
- Reject non-HTTP(S) schemes.

## Admin API

All routes require authenticated session (or existing API auth pattern used by other user-scoped CRUD). Resources always filtered by `user_id`.

| Method | Path | Behavior |
|--------|------|----------|
| GET | `/api/webhooks` | List; query `domainId` optional |
| POST | `/api/webhooks` | Create; body: name, url, events[], domainId?, enabled; response includes `secret` once |
| PATCH | `/api/webhooks/:id` | Update name, url, events, enabled, domainId (not secret) |
| POST | `/api/webhooks/:id/rotate-secret` | New secret, return once |
| DELETE | `/api/webhooks/:id` | Delete |
| POST | `/api/webhooks/:id/test` | Enqueue a **synthetic** test delivery (see below) |
| GET | `/api/webhook-deliveries` | List with filters: status, webhookId, eventType, limit |
| POST | `/api/webhook-deliveries/:id/replay` | Replay dead/success/pending (re-queue per replay rules) |

**Test delivery:** Does not require a real `send_events` row. Use `send_event_id = 0` (allowed only for test deliveries; document as sentinel). Payload shape matches production with:

- `data.test: true`
- `data.message_id: "mh-test"`
- `data.send_event_id: 0`
- `data.status` / `type` from the webhook’s first subscribed event, or `sent` / `email.sent` if all three are subscribed
- Placeholder from/to/subject/domain from the user’s first domain when available, else fixed examples

Unique key for test rows: still `(webhook_id, send_event_id, event_type)` — concurrent tests of the same event on the same webhook may hit the unique constraint; API should **reuse** the existing test delivery row and reset it to pending (replay semantics) instead of failing.

Validation: URL format + SSRF check on create/update; events non-empty subset of allowed three; domainId must belong to user when set.

## UI

### Global Webhooks page (replace Placeholder)

- Table: name, scope (Account / domain name), URL (truncated), events tags, enabled switch, last delivery status snippet
- Create / edit drawer
- After create / rotate: modal with full secret + copy (warning: shown once)
- Actions: test, open deliveries filtered by webhook, delete
- Deliveries panel or sub-view: status, event, attempts, HTTP code, error, replay button

### Domain detail → Webhooks tab

- Same CRUD scoped to current `domainId` (override endpoints)
- Short help text: “If this domain has any enabled endpoints for an event, account-level endpoints for that event are skipped.”

Reuse existing design system: PageHeader, SectionCard, StatusPill, CodeBlock for secret / sample payload docs.

## Code Modules

| Module | Responsibility |
|--------|----------------|
| `src/webhook-model.js` | Pure: event map, resolve list, payload build, sign, backoff, URL validation helpers |
| `src/webhook-dispatcher.js` | Worker loop, fetch, SSRF resolve checks, status updates |
| `src/db.js` | Schema, CRUD, enqueue helper |
| `src/server.js` | Routes; call enqueue after terminal status transitions |
| `src/pages/Webhooks.tsx` (+ small components) | Global UI |
| Domain detail tab | Wire domain-scoped list |

## Testing

- Unit: resolve rules (domain override vs account), signature stable, backoff schedule, idempotent enqueue key, event type mapping
- DB: CRUD isolation by user; unique delivery constraint
- Dispatcher: mock fetch—2xx success; 500 retries; timeout; SSRF rejected URL never fetched
- Server API: auth boundary; secret not returned on list

## Delivery Order

1. Schema + db CRUD + model pure functions + tests  
2. Enqueue hooks on terminal status  
3. Dispatcher worker + SSRF  
4. REST API  
5. Global Webhooks UI  
6. Domain detail tab  
7. i18n + build gate  

## Success Criteria

- Terminal status change creates at most one delivery row per webhook endpoint  
- Domain override suppresses account endpoints for that event  
- Failed POSTs remain `pending` (or become `dead`) with backoff and appear in UI  
- Manual replay works  
- Worker claim uses `processing` lease to avoid double POST  

- Secrets encrypted at rest; shown once on create/rotate  
- `npm test` / `npm run build` pass  

## Open Follow-ups (out of MVP)

- `queued` / `deferred` events  
- Multiple concurrent endpoints without override (always merge account+domain)  
- Dead-letter alerts / email to owner  
- Signing key versioning (`v2=`)  
- OpenAPI doc page for receivers