One hub for the background knowledge, SOPs, and Codex prompts.
Alert Emails and SMS
Mission Control sends alert notifications from cloud services only. A Cloudflare Worker runs every 15 minutes, reads the live dashboard APIs, compares the active alert set to the last sent fingerprint in Workers KV, and posts changed alert state to a Google Apps Script relay. The relay sends through the Local Reach Gmail account to the configured email and SMS gateway recipients.
Cloud pieces involved
Dashboard APIs: Cloudflare Pages Functions expose sanitized Mission Control data from R2, including /api/snapshot and /api/wp-malware-scans.
Alert Worker:mission-control-alerts runs on Cloudflare Cron Triggers every 15 minutes and evaluates only the live Pages APIs.
Health datapoints: the dashboard Backup Integrity tile shows per-site B2/restic website backup integrity, CloudPanel app-state backup status, and a separate ALERTS: clear line from /api/alert-system-health.
Deduping: Workers KV stores the current alert fingerprint so repeated checks do not send the same email/SMS every 15 minutes.
Email relay: Google Apps Script validates a shared secret stored outside Git, allowlists the configured recipients, and sends via MailApp.
Sender: Messages appear from the Local Reach Gmail account with display name Mission Control and reply-to set to that mailbox.
When a message is sent
Dashboard API failure:/api/snapshot cannot be read, or /api/wp-malware-scans cannot be read.
Collector stale: the snapshot timestamp is missing or older than the configured stale threshold, currently 30 minutes.
Load Balancer hard failure: Cloudflare LB status cannot be read, no LB pools are healthy, or the active pool is down.
Load Balancer side down: WEST or EAST is unhealthy/degraded for at least 15 minutes, including primary-down failover or standby-side loss while the primary still serves traffic.
HTTP failure: an enabled site's home or deep HTTP check is not OK.
Customer-facing page failure: a public page returns a tiny/blank body, lacks expected site identity text, shows known fatal/error-page text, or includes known suspicious public HTML indicators.
SSL urgent: a public edge or origin certificate has fewer than 7 days remaining.
TLS handshake failure: a public or origin certificate check errors even when no valid day count is available.
Backups: a website backup row reports ok: false, a backup is older than the configured maximum age, currently over 8 days, backup integrity reports ok: false, or a WEST/EAST CloudPanel app-state backup target fails or becomes stale.
Malware scans: any WordPress malware scan row is not pass, has a scan error, or the malware scan API returns no site rows.
What malware review means
Public HTML: suspicious external script domains, known bad markers, obfuscated JavaScript patterns, unexpected redirects, and public/origin mismatches can make a site need review.
Database indicators: options, posts, postmeta, theme mods, widgets, snippet/plugin records, header/footer injection areas, and practical Avada custom-code areas are checked without publishing full payloads.
Admin/code drift: new admin users and active-theme PHP drift are review triggers. Routine plugin/theme file changes are shown as observed churn unless the scanner promotes them to review.
Uploads PHP: known benign blocked artifacts are not described with scary wording. Review is reserved for drift, executable PHP, suspicious files, or unknown PHP in uploads.
What the PAGE signal catches
Blank or tiny output: a site can return HTTP 200 while serving an empty, placeholder, or broken page.
Expected-content drift: each site has one or more safe brand/content markers. If none appear, Mission Control treats the page as customer-impacting.
Fatal/error pages: WordPress critical errors, database connection errors, PHP fatal/parse errors, install screens, and Cloudflare origin errors are flagged.
Injected public HTML signals: known incident markers such as suspicious /click?key, .icu URLs, and obfuscated JavaScript patterns are flagged without storing the full payload.
What does not send a message
Green dashboard state and normal collector runs do not send anything.
The Worker does not send an all-clear email; it silently resets KV to clear when no alerts remain.
Maintenance-overdue items do not currently trigger email/SMS.
Ambiguous collector counters without a concrete site or service failure do not currently trigger email/SMS.
Short Load Balancer side blips under 15 minutes do not trigger email/SMS unless they become a hard failure.
Routine plugin/theme churn by itself does not trigger email/SMS unless the malware scanner marks the site for review.
How expected alerts get cleared
Fix the source condition: restore the site/API/backup/SSL/scanner condition that caused the alert, then let the next Worker check reset the state.
Expected website work: when plugin updates, theme edits, or new users are intentional, review the scanner row, confirm the trigger, and clear or baseline it in the scanner data rather than ignoring it.
Future investigations: the expected workflow is to identify what triggered the concern, decide whether it is actually risky, ask for any needed context, then recommend either further action or clearing the alert.
Monthly live alert path test
The maintenance row Monthly alert path test (email/SMS) is the scheduled end-to-end test for the alert system.
The test intentionally causes a temporary LRWD HTTP page failure, confirms email/SMS receipt, restores the origin, and verifies the alert Worker clears based on the next collector input.
Do not make /lb-health.txt fail, disable shared Cloudflare pools, or alter the shared LB monitor for this test because that health path affects multiple managed sites.
Trusted Devices and WARP Access
Mission Control is protected by Cloudflare Access. Trusted LocalReach WARP devices can bypass the email-code challenge when the Access application has the reusable Bypass LocalReach WARP Devices policy attached.
How the bypass works
WARP must be connected: Cloudflare has to see the request as coming from the LocalReach Zero Trust organization.
The device must satisfy posture: the reusable policy requires the Cloudflare Warp posture check. A normal browser session without a matching trusted WARP device still sees the email-code login.
Mission Control app policy: the Mission Control Access app uses the trusted-device bypass so reviewed WARP devices can open the dashboard without re-entering the email code.
Email login remains available: users who do not match the trusted-device bypass still have to pass the configured Cloudflare Access login policy.
What the dashboard tracks
HTTP Checks datapoint: the main dashboard shows X/X Trusted devices inside the existing HTTP Checks block.
Count meaning: the first number is reviewed trusted devices; the second number is total reviewed devices in Mission Control's trusted-device inventory.
Current limitation: this is a reviewed inventory from Cloudflare Zero Trust, not live API inventory. The available Mission Control tokens do not currently have Zero Trust device-read permission.
When to review it
Review the trusted-device inventory whenever a Mac, phone, or other WARP device is added, retired, lost, or transferred.
If a device should no longer bypass Mission Control, remove or disable that device in Cloudflare Zero Trust and update the Mission Control trusted-device inventory.
If live inventory is needed later, create a scoped Cloudflare token with Zero Trust read permissions and store it only in protected runner configuration, never in Git or iCloud.
Server access and break-glass SSH
Routine server access: use mc-ops-01-warp, west-01-warp, and east-01-warp. These aliases go through Cloudflare Access SSH.
Break-glass fallback: direct public-IP SSH remains available with key auth only so the servers can still be reached if WARP, Access, or a Cloudflare tunnel fails.
What to do if WARP fails: connect with the direct alias only long enough to repair cloudflared, Cloudflare Access policy, sshd, or ufw, then return to the WARP aliases.
Public exposure baseline: WEST/EAST keep public 22, 80, and 443. mc-ops-01 keeps public 22. Password SSH is disabled, and fail2ban is active.
Closed management ports: raw CloudPanel :8443, FTP 21, mail 25, app 3000, and database 3306 should remain closed publicly. CloudPanel access should use the protected Access hostnames.
Website Backups
B2/restic backup inventory with manual backup commands and emergency restore shortcuts.
Shared task queue for Mission Control work. Add items, edit them later, move them up or down the stack, and mark them complete with an automatic completion timestamp.