# MailHub Delivery Webhooks Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Implement signed HTTPS webhooks for terminal email events (`sent` / `bounced` / `failed`) with account default + domain override, multi-URL subscriptions, and an observable SQLite delivery queue (claim, retry, logs, replay).

**Architecture:** Pure helpers in `webhook-model.js`; CRUD + enqueue in `db.js`; HTTP worker in `webhook-dispatcher.js` started from `server.js`; REST under `/api/webhooks` and `/api/webhook-deliveries`; React pages replace Placeholder and domain Webhooks tab. Secrets use existing `encryptSecret` / `decryptSecret`.

**Tech Stack:** Node.js ESM, SQLite (`node:sqlite`), React + Ant Design admin UI, `node:test`

**Spec:** `docs/superpowers/specs/2026-07-09-mailhub-webhooks-design.md`

---

## File map

| File | Responsibility |
|------|----------------|
| Create `src/webhook-model.js` | Event map, resolve targets, payload, HMAC sign, backoff, URL precheck helpers (pure) |
| Create `src/webhook-dispatcher.js` | Worker loop, claim, SSRF-safe fetch, status updates |
| Create `test/webhook-model.test.js` | Pure model unit tests |
| Create `test/webhook-dispatcher.test.js` | Dispatcher with mocked fetch |
| Modify `src/db.js` | Tables, webhook CRUD, enqueue, claim, replay, list deliveries |
| Modify `src/server.js` | API routes; start worker; ensure enqueue after terminal writes if not fully inside db |
| Create `src/pages/Webhooks.tsx` | Global webhooks + deliveries UI |
| Modify `src/frontend/App.tsx` | Wire Webhooks page, load webhooks data |
| Modify `src/pages/Domains/DomainDetail.tsx` | Real Webhooks tab |
| Modify `src/frontend/types.ts` | Webhook / delivery types |
| Modify `src/frontend/services/api.ts` | Client methods |
| Modify `src/frontend/i18n/index.js` | zh/en strings |
| Modify `test/frontend-i18n.test.js` | New keys if asserted |
| Create `test/server-webhooks-api.test.js` | API auth + CRUD smoke (follow `server-admin-api.test.js` patterns) |

---

### Task 1: Pure webhook model

**Files:**
- Create: `src/webhook-model.js`
- Create: `test/webhook-model.test.js`

- [ ] **Step 1: Write failing tests**

```js
// test/webhook-model.test.js
import assert from 'node:assert/strict';
import { test } from 'node:test';
import {
  TERMINAL_WEBHOOK_EVENTS,
  eventTypeForStatus,
  resolveWebhooksForEvent,
  buildWebhookPayload,
  signWebhookBody,
  nextBackoffMs,
  isTerminalWebhookStatus
} from '../src/webhook-model.js';

test('maps terminal statuses to email.* types', () => {
  assert.equal(eventTypeForStatus('sent'), 'email.sent');
  assert.equal(eventTypeForStatus('bounced'), 'email.bounced');
  assert.equal(eventTypeForStatus('failed'), 'email.failed');
  assert.equal(eventTypeForStatus('queued'), null);
});

test('domain webhooks override account for the same event', () => {
  const account = [
    { id: 1, domainId: null, enabled: true, events: ['sent', 'failed'] },
    { id: 2, domainId: null, enabled: true, events: ['bounced'] }
  ];
  const domain = [
    { id: 3, domainId: 9, enabled: true, events: ['sent'] }
  ];
  const resolved = resolveWebhooksForEvent({
    accountWebhooks: account,
    domainWebhooks: domain,
    eventType: 'sent'
  });
  assert.deepEqual(resolved.map((w) => w.id), [3]);
});

test('falls back to account when domain has no matching enabled subscription', () => {
  const resolved = resolveWebhooksForEvent({
    accountWebhooks: [{ id: 1, domainId: null, enabled: true, events: ['failed'] }],
    domainWebhooks: [{ id: 3, domainId: 9, enabled: true, events: ['sent'] }],
    eventType: 'failed'
  });
  assert.deepEqual(resolved.map((w) => w.id), [1]);
});

test('signs body with Stripe-style t and v1', () => {
  const body = '{"id":"whd_1"}';
  const header = signWebhookBody(body, 'secret', 1_700_000_000);
  assert.equal(header.startsWith('t=1700000000,v1='), true);
  assert.match(header, /^t=\d+,v1=[0-9a-f]{64}$/);
});

test('backoff grows then caps', () => {
  assert.ok(nextBackoffMs(1) < nextBackoffMs(2));
  assert.equal(nextBackoffMs(10), nextBackoffMs(20));
});
```

- [ ] **Step 2: Run tests — expect FAIL**

Run: `node --test test/webhook-model.test.js`

- [ ] **Step 3: Implement `src/webhook-model.js`**

Export at least:

```js
export const TERMINAL_WEBHOOK_EVENTS = ['sent', 'bounced', 'failed'];
export const MAX_WEBHOOK_ATTEMPTS = 8;
export const WEBHOOK_LEASE_MS = 2 * 60 * 1000;

export function isTerminalWebhookStatus(status) {
  return TERMINAL_WEBHOOK_EVENTS.includes(status);
}

export function eventTypeForStatus(status) {
  if (status === 'sent') return 'email.sent';
  if (status === 'bounced') return 'email.bounced';
  if (status === 'failed') return 'email.failed';
  return null;
}

/** @param {{ accountWebhooks: any[]; domainWebhooks: any[]; eventType: string }} input */
export function resolveWebhooksForEvent({ accountWebhooks, domainWebhooks, eventType }) {
  const matches = (list) => (list || []).filter(
    (w) => w.enabled !== false && w.enabled !== 'false' && Array.isArray(w.events) && w.events.includes(eventType)
  );
  const domainHits = matches(domainWebhooks);
  if (domainHits.length) return domainHits;
  return matches(accountWebhooks);
}

export function buildWebhookPayload({ deliveryId, eventType, createdAt, sendEvent, test = false }) {
  const status = sendEvent.status;
  return {
    id: `whd_${deliveryId}`,
    type: eventTypeForStatus(status) || eventType,
    created_at: createdAt,
    data: {
      ...(test ? { test: true } : {}),
      message_id: test ? 'mh-test' : `mh-${sendEvent.id}`,
      send_event_id: sendEvent.id,
      queue_id: sendEvent.queueId || '',
      status,
      domain: sendEvent.domain || '',
      from: sendEvent.sender || '',
      to: sendEvent.recipients || [],
      subject: sendEvent.subject || '',
      detail: sendEvent.detail || '',
      delivered_at: sendEvent.deliveredAt || null
    }
  };
}

export function signWebhookBody(rawBody, secret, unixSeconds = Math.floor(Date.now() / 1000)) {
  const crypto = awaitImportOrRequireCrypto(); // use import crypto from 'node:crypto' at top
  const signed = `${unixSeconds}.${rawBody}`;
  const v1 = crypto.createHmac('sha256', secret).update(signed).digest('hex');
  return `t=${unixSeconds},v1=${v1}`;
}

/** attemptCount after increment for failed path; attempt 1 → 60s, … cap 12h */
export function nextBackoffMs(attemptCount) {
  const table = [60_000, 300_000, 1_800_000, 7_200_000, 21_600_000, 43_200_000];
  const index = Math.max(0, Math.min(table.length - 1, attemptCount - 1));
  return table[index];
}
```

Use top-level `import crypto from 'node:crypto'`. Fix any pseudocode above to real ESM.

Also export `parseWebhookEventsJson`, `normalizeWebhookEvents(input)` validating non-empty subset of three events.

- [ ] **Step 4: Run tests — PASS**

Run: `node --test test/webhook-model.test.js`

- [ ] **Step 5: Commit**

```bash
git add src/webhook-model.js test/webhook-model.test.js
git commit -m "feat(webhooks): add pure model for events, resolve, sign, backoff"
```

---

### Task 2: Schema + DB CRUD + enqueue

**Files:**
- Modify: `src/db.js`
- Create: `test/webhook-db.test.js` (or extend `test/db.test.js` if preferred — prefer dedicated file)

- [ ] **Step 1: Failing tests for isolation and enqueue idempotency**

```js
// test/webhook-db.test.js — follow openTempDb / createUser patterns from test/db.test.js
test('creates webhooks per user and lists by domain scope', () => { /* ... */ });
test('enqueueWebhookDeliveries is idempotent per webhook+event+send_event', () => { /* ... */ });
test('domain override skips account webhooks for that event', () => { /* ... */ });
test('secret not returned on list; create returns plaintext once', () => { /* ... */ });
```

- [ ] **Step 2: Run — FAIL**

- [ ] **Step 3: Schema in `initDb` / migrations block**

```sql
CREATE TABLE IF NOT EXISTS webhooks (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  user_id INTEGER NOT NULL,
  domain_id INTEGER,
  name TEXT NOT NULL,
  url TEXT NOT NULL,
  secret_ciphertext TEXT NOT NULL,
  secret_prefix TEXT NOT NULL,
  events_json TEXT NOT NULL,
  enabled TEXT NOT NULL DEFAULT 'true',
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS webhook_deliveries (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  webhook_id INTEGER NOT NULL,
  user_id INTEGER NOT NULL,
  send_event_id INTEGER NOT NULL,
  event_type TEXT NOT NULL,
  payload_json TEXT NOT NULL,
  status TEXT NOT NULL,
  attempt_count INTEGER NOT NULL DEFAULT 0,
  next_attempt_at TEXT NOT NULL,
  last_attempt_at TEXT,
  response_status INTEGER,
  response_body_preview TEXT NOT NULL DEFAULT '',
  error TEXT NOT NULL DEFAULT '',
  created_at TEXT NOT NULL,
  FOREIGN KEY(webhook_id) REFERENCES webhooks(id) ON DELETE CASCADE,
  UNIQUE(webhook_id, send_event_id, event_type)
);

CREATE INDEX IF NOT EXISTS idx_webhooks_user_id ON webhooks(user_id);
CREATE INDEX IF NOT EXISTS idx_webhooks_user_domain ON webhooks(user_id, domain_id);
CREATE INDEX IF NOT EXISTS idx_webhook_deliveries_status_next ON webhook_deliveries(status, next_attempt_at);
CREATE INDEX IF NOT EXISTS idx_webhook_deliveries_user_created ON webhook_deliveries(user_id, created_at);
```

No FK on `send_event_id` (test sentinel `0`).

- [ ] **Step 4: Implement API functions**

- `listWebhooks(userId, { domainId }?)`
- `createWebhook(userId, { name, url, events, domainId, enabled })` → returns row + `secret` plaintext once; store `encryptSecret(secret)`, prefix first 8 chars
- `updateWebhook(userId, id, patch)`
- `rotateWebhookSecret(userId, id)` → new secret once
- `deleteWebhook(userId, id)`
- `enqueueWebhookDeliveries(sendEvent)` — see **Enqueue contract** below.
- `claimWebhookDeliveries(limit)` — transaction: select pending due, set processing + lease `next_attempt_at`; return claimed rows **already joined** with webhook `url` and **decrypted** secret (decrypt only inside `db.js`; do **not** export `encryptSecret`/`decryptSecret`). Prefer shape: `{ delivery, webhook: { id, url, secret } }`.
- `getWebhookDeliveryTarget(deliveryId)` optional alternative if claim returns thin rows — still decrypt only in db.js.
- `reapExpiredWebhookProcessing()`
- `completeWebhookDeliverySuccess(id, { responseStatus, bodyPreview })` — set `attempt_count = attempt_count + 1`, status success
- `completeWebhookDeliveryFailure(id, { responseStatus, bodyPreview, error })` — set `attempt_count = attempt_count + 1`; if `attempt_count < 8` then status pending + `next_attempt_at = now + nextBackoffMs(attempt_count)`; else dead
- `listWebhookDeliveries(userId, filters)`
- `replayWebhookDelivery(userId, id)` — reject if processing
- `enqueueWebhookTestDelivery(userId, webhookId)` — send_event_id 0, reuse unique key via reset-to-pending

#### Enqueue contract

Canonical input (all fields preferred; helper may fill gaps):

```js
{
  id,              // send_events.id (required)
  userId,          // required; no-op if null/undefined
  domainId,        // may be null
  domain,          // optional display name; if missing, JOIN domains by domainId
  status,          // sent|bounced|failed
  sender,
  recipients,      // array
  subject,
  detail,
  queueId,
  deliveredAt
}
```

**Idempotent insert (critical):** For each resolved webhook, do **not** blindly `INSERT OR IGNORE` then `UPDATE` by `lastInsertRowid()`.

Preferred pattern inside a transaction per webhook (or one transaction for all):

1. `SELECT id FROM webhook_deliveries WHERE webhook_id=? AND send_event_id=? AND event_type=?`
2. If row exists → **skip** (leave payload/status untouched)
3. Else `INSERT` with placeholder `payload_json='{}'`, read `lastInsertRowid()` only when `result.changes === 1`
4. `UPDATE` that id with final `payload_json` from `buildWebhookPayload` **in the same transaction**

Alternatively: try `INSERT` final payload in one statement using a pre-allocated id strategy only if SQLite version allows; simplest is insert+update on **new** rows only (`changes === 1`).

#### Wire points

1. **`logSendEvent`**: after successful insert, if terminal status, call `enqueueWebhookDeliveries({ ...event, id: lastInsertRowid, recipients: normalized array, domain: resolved name })` inside try/catch (log errors; never throw to caller).
2. **`updateSendEventDelivery`**: when `nextStatus` is terminal **and** `nextStatus !== row.status`, load user_id/domain/recipients from row, resolve domain name, enqueue with new status/detail/deliveredAt; try/catch.

Keep enqueue try/catch logged so webhook failure never breaks mail path.

- [ ] **Step 5: Tests PASS + commit**

```bash
git add src/db.js test/webhook-db.test.js
git commit -m "feat(webhooks): add schema, CRUD, and terminal enqueue hooks"
```

---

### Task 3: Dispatcher worker + SSRF

**Files:**
- Create: `src/webhook-dispatcher.js`
- Create: `test/webhook-dispatcher.test.js`
- Modify: `src/server.js` (start/stop worker on listen)

- [ ] **Step 1: Tests with injectable `fetch` and clock**

```js
test('posts signed body and marks success on 2xx', async () => { /* mock fetch ok */ });
test('schedules retry on 500', async () => { /* */ });
test('marks dead after max attempts', async () => { /* */ });
test('rejects private IP targets without calling fetch', async () => { /* */ });
```

- [ ] **Step 2: Implement dispatcher**

```js
// Core loop
export function startWebhookWorker({ intervalMs = 10_000, batchSize = 3, fetchImpl = fetch } = {}) { ... }
export function stopWebhookWorker() { ... }
export async function processWebhookBatch({ fetchImpl, batchSize } = {}) {
  reapExpiredWebhookProcessing();
  const rows = claimWebhookDeliveries(batchSize);
  for (const row of rows) {
    await deliverOne(row, fetchImpl);
  }
}
```

`deliverOne` (row already includes url + decrypted secret from db claim/join):

1. If url/secret missing → complete as dead/error
2. Validate URL (https; optional `WEBHOOK_ALLOW_HTTP_LOCAL` only for loopback HTTP)
3. DNS lookup / block private IPv4 **and** IPv6 ULA/link-local/loopback (fail closed); never call fetch on blocked targets
4. `fetch(url, { method: 'POST', headers: { 'Content-Type': 'application/json', 'User-Agent': 'MailHub-Webhook/1.0', 'X-MailHub-Signature': sign..., 'X-MailHub-Event': type, 'X-MailHub-Delivery': id }, body: payload_json, signal: AbortSignal.timeout(10_000), redirect: 'manual' })`
5. Success / failure complete helpers (each finished HTTP attempt increments `attempt_count` once)

- [ ] **Step 3: Start worker at module bootstrap** (same pattern as `startDnsAutoChecker` / delivery tracker — not only inside `listen`). Optionally skip when `WEBHOOK_WORKER_ENABLED=0` for tests.

- [ ] **Step 4: Tests PASS + commit**

```bash
git add src/webhook-dispatcher.js test/webhook-dispatcher.test.js src/server.js
git commit -m "feat(webhooks): add delivery worker with claim and SSRF guards"
```

---

### Task 4: REST API

**Files:**
- Modify: `src/server.js`
- Create: `test/server-webhooks-api.test.js`

- [ ] **Step 1: Tests** — session auth required; user A cannot see user B webhooks; create returns secret; list omits secret; test + replay endpoints

Follow patterns in `test/server-admin-api.test.js` (login cookie, temp data dir).

- [ ] **Step 2: Routes** (session user required, same as other user APIs)

```
GET    /api/webhooks
POST   /api/webhooks
PATCH  /api/webhooks/:id
POST   /api/webhooks/:id/rotate-secret
DELETE /api/webhooks/:id
POST   /api/webhooks/:id/test
GET    /api/webhook-deliveries
POST   /api/webhook-deliveries/:id/replay
```

Validate body; map errors to 400.

- [ ] **Step 3: Tests PASS + commit**

```bash
git add src/server.js test/server-webhooks-api.test.js
git commit -m "feat(webhooks): expose management and delivery APIs"
```

---

### Task 5: Frontend API + types + i18n

**Files:**
- Modify: `src/frontend/types.ts`
- Modify: `src/frontend/services/api.ts`
- Modify: `src/frontend/i18n/index.js`
- Modify: `test/frontend-i18n.test.js` (if new keys asserted)

- [ ] **Step 1: Types**

```ts
export type WebhookEvent = 'sent' | 'bounced' | 'failed';
export interface Webhook {
  id: number;
  userId: number;
  domainId: number | null;
  name: string;
  url: string;
  secretPrefix: string;
  events: WebhookEvent[];
  enabled: boolean;
  createdAt: string;
  updatedAt: string;
  secret?: string; // only on create/rotate responses
}
export interface WebhookDelivery {
  id: number;
  webhookId: number;
  sendEventId: number;
  eventType: string;
  status: 'pending' | 'processing' | 'success' | 'dead';
  attemptCount: number;
  // ...
}
```

- [ ] **Step 2: api.ts methods** mirroring REST

- [ ] **Step 3: i18n keys** for page chrome (zh-CN + en-US): titles, events labels, empty states, secret warning, replay, test

- [ ] **Step 4: Commit**

```bash
git add src/frontend/types.ts src/frontend/services/api.ts src/frontend/i18n/index.js test/frontend-i18n.test.js
git commit -m "feat(webhooks): add frontend types, API client, and i18n"
```

---

### Task 6: Global Webhooks page + App wiring

**Files:**
- Create: `src/pages/Webhooks.tsx`
- Modify: `src/frontend/App.tsx`
- Optionally: small components under `src/components/webhook/`

- [ ] **Step 1: Build `Webhooks.tsx` with optional `domainId?: number` prop**

Use PageHeader, SectionCard, StatusPill, Table, Drawer/Form, Modal for secret once, Popconfirm delete, CodeBlock for sample receiver docs.

When `domainId` is set: hide global page chrome / filter list to that domain; pre-fill create form scope.

Sections:

1. Endpoints table + create button (columns: name, scope, URL truncated, events, enabled, **last delivery snippet**)  
2. Deliveries table (filter by webhook) + replay  

Handlers call `api.*` and refresh list. Toggle enabled via PATCH.

- [ ] **Step 2: App.tsx**

- Import Webhooks page  
- Replace Placeholder for `webhooks` view  
- Load webhooks (and optionally deliveries) in `loadAll` or lazy on view enter  

- [ ] **Step 3: `npx tsc --noEmit` PASS**

- [ ] **Step 4: Commit**

```bash
git add src/pages/Webhooks.tsx src/frontend/App.tsx src/components/webhook
git commit -m "feat(webhooks): add global webhooks management UI"
```

---

### Task 7: Domain detail Webhooks tab

**Files:**
- Modify: `src/pages/Domains/DomainDetail.tsx`
- Possibly reuse list component with `domainId` prop from Task 6

- [ ] **Step 1: Replace Placeholder tab with filtered Webhooks panel** (`domainId={domain.id}`) + help Alert about override semantics

- [ ] **Step 2: tsc PASS + commit**

```bash
git add src/pages/Domains/DomainDetail.tsx src/pages/Webhooks.tsx
git commit -m "feat(webhooks): wire domain-scoped webhook overrides in detail tab"
```

---

### Task 8: Full gate

**Files:** polish only if needed

- [ ] **Step 1:** `npm test` — all pass  
- [ ] **Step 2:** `npm run build` — pass; commit built `public/` assets if hashes change  
- [ ] **Step 3:** Manual checklist  
  - Create account webhook, copy secret  
  - Send test  
  - Force fail (bad URL) → pending retry fields  
  - Replay  
  - Domain override skips account for same event  
- [ ] **Step 4:** Final commit if assets/docs changed  

```bash
git add -A
git commit -m "chore(webhooks): build assets after webhook feature"
```

---

## Notes

- Do not block mail send on webhook errors.  
- Never log full secrets.  
- Align field naming with existing API camelCase JSON helpers in `db.js` / server.  
- Follow `encryptSecret` usage from DNS credentials.  
- After implementation, use @superpowers:verification-before-completion before claiming done.  
- Deploy only after user requests per Agents.md.

## Execution handoff

Plan complete. Choose:

1. **Subagent-Driven (recommended)** — fresh subagent per task + review  
2. **Inline Execution** — this session with checkpoints