Mission Control Runbooks

Maintenance overview

What it means: These are scheduled tasks that prevent surprises (patches, backups, failover drills).
Status colors: Green = not due soon. Yellow = due soon (inside warning window). Red = overdue.
Due math: next due = last done + cadence. “Due in” shows days remaining (negative = overdue).
If you see RED: Open the matching runbook below and do the steps. Update the collector config on TrueNAS at /mnt/Piscine/infra/mission-control-collector/repo/config/maintenance.json when done (this file is not in the Mission Control UI repo).

Advanced notes:

Cadence and warn windows are defined per item in /mnt/Piscine/infra/mission-control-collector/repo/config/maintenance.json (TrueNAS).
Status is worst-case across items. Runbook links are anchored to the IDs in that file.

Find the item in the dashboard (Maintenance table) and click its Runbook link.
Follow the step-by-step list below for that item.
Verify success (checks included per item).
Update /mnt/Piscine/infra/mission-control-collector/repo/config/maintenance.json with the new last_done_at, result, and notes, then rerun the collector.

Choose the right restore method (Website backups vs Server backups vs Snapshots) + copyable Copilot prompts.

Website backups live on TrueNAS at /mnt/Piscine/Home/1. Local Reach Web Design/Automated Backups/website-backups/<domain>/<YYYY-MM-DD> (HST date).
Weekly schedule: Sundays 03:30 UTC via /mnt/Piscine/infra/backup-websites/run-weekly.sh.
Included domains: localreachwebdesign.com, laspanishlessons.com, clsps.net, kitsapcriminaldefense.com, dukeplumb.com, traveltechus.com, blog.traveltechus.com, majorchangeinitiative.org, blog.majorchangeinitiative.org.
Mission Control backup badges are sourced from backup-status.json and show OK when a backup is within 7 days.

What this is: Apply OS/package updates and reboot if needed.
Why it matters: Security fixes and stability; missing patches can leave vulnerabilities.
When it’s due: Every 30 days (warn inside 7 days).
Checklist:
- Open Runbook Prompts (West/East patch + reboot)
- SSH to each server (west-01, east-01).
- Run apt update && apt upgrade (or the host’s package manager).
- If prompted, schedule/perform reboot during low traffic.
- After reboot, verify key sites load (LRWD, traveltechus).
Confirm success: No pending critical updates; dashboard server tile shows services OK and no reboot required.
Troubleshooting:
- If a package is held/broken: check /var/log/apt, consider apt --fix-broken install.
- If services fail after update: check systemctl status <service>, review logs, restart.

What this is: A pragmatic WordPress hygiene scan run over SSH against each site’s origin.
What it includes: Core checksum verification (WP-CLI), plugin/theme inventory, admin list, “PHP in uploads” check, heuristic grep sweep, optional ClamAV signature scan.
What it does not do: No auto-cleanup, no quarantine, no deletes. It produces indicators and an auditable report.
Where reports live (UI): WordPress Malware Scans (Access-protected report downloads).
Where reports live (TrueNAS): /mnt/Piscine/infra/mission-control-collector/artifacts/wp-malware-scan/<site_id>/<YYYY-MM>/
How to run (from your Mac):
- ssh -t truenas "/mnt/Piscine/infra/mission-control-collector/run-wp-malware-scans.sh"
- ssh -t truenas "/mnt/Piscine/infra/mission-control-collector/run-wp-malware-scans.sh --only-site-id kitsap" (single site)
Confirm success: UI shows a fresh “Generated” timestamp and each site is pass or needs_review with counts.
If something is flagged: Take a snapshot/backup first, then do targeted investigation (identify the file, timeframe, and whether it matches a known maintenance window).

What this is: Restore a backup/snapshot to staging and ensure it works.
Why it matters: Proves backups are usable; avoids surprises during real incidents.
When it’s due: Every 90 days (warn inside 7 days).
Checklist:
- Pick the latest HST-dated website backup for a key site (e.g., LRWD or traveltechus).
- Restore to a staging environment or new VM.
- Update hosts/DB config as needed for staging.
- Load the site and run a basic sanity check (homepage, login if applicable).
- Record time-to-restore and any issues.
Confirm success: Restored site loads in staging, data intact, and you documented restore time.
Troubleshooting:
- Backup won’t import: check archive integrity, try a different snapshot.
- Permissions/config issues: fix file ownership, update env/DB strings for staging.

Purpose: Karen works locally with Codex and feature branches; Nick deploys when ready.

Repo: localreachwd/mci-site Local preview: http://localhost:3000 Server preview: mci.localreachwebdesign.com (CloudPanel + PM2)

Sync and start dev: git pull, npm ci (if needed), npm run dev, open http://localhost:3000.
Edit locally with Codex and refresh the browser to validate changes.
Push a branch and tell Nick the branch name + what changed.

What this is: Practice failing all LB-backed traffic from West to East and back.
Why it matters: Ensures the load balancer + origins really fail over when needed.
When it’s due: Every 90 days (warn inside 7 days).
Checklist:
- During a quiet window, drain/disable West pool in Cloudflare LB.
- Confirm traffic serves from East (dashboard LB tile should show Active: lrwd-east).
- Run basic site checks from East for localreachwebdesign.com, clsps.net, kitsapcriminaldefense.com, and dukeplumb.com.
- Re-enable West and confirm Active returns to lrwd-west.
- Document any issues or slow steps.
Confirm success: LB shows primary healthy after re-enable; site loads fine from both regions.
Troubleshooting:
- If LB won’t fail over: check origin health, SSL, and DNS settings in Cloudflare.
- If East is slow/down: inspect east-01 services (nginx/php/mysql/redis) and logs.