Mission Control Runbooks

Maintenance overview

• What it means: These are scheduled tasks that prevent surprises (patches, backups, failover drills).
• Status colors: Green = not due soon. Yellow = due soon (inside warning window). Red = overdue.
• Due math: next due = last done + cadence. “Due in” shows days remaining (negative = overdue).
• If you see RED: Open the matching runbook below and do the steps. Update the collector config on TrueNAS at /mnt/Piscine/infra/mission-control-collector/repo/config/maintenance.json when done (this file is not in the Mission Control UI repo).

What to do when something is RED

• Find the item in the dashboard (Maintenance table) and click its Runbook link.
• Follow the step-by-step list below for that item.
• Verify success (checks included per item).
• Update /mnt/Piscine/infra/mission-control-collector/repo/config/maintenance.json with the new last_done_at, result, and notes, then rerun the collector.

Backup & Restore SOP

Choose the right restore method (Website backups vs Server backups vs Snapshots) + copyable Copilot prompts.

• Website backups live on TrueNAS at /mnt/Piscine/Home/1. Local Reach Web Design/Automated Backups/website-backups/<domain>/<YYYY-MM-DD> (HST date).
• Weekly schedule: Sundays 03:30 UTC via /mnt/Piscine/infra/backup-websites/run-weekly.sh.
• Included domains: localreachwebdesign.com, laspanishlessons.com, clsps.net, traveltechus.com, blog.traveltechus.com, majorchangeinitiative.org, blog.majorchangeinitiative.org.
• Mission Control backup badges are sourced from backup-status.json and show OK when a backup is within 7 days.

Monthly patch window (OS + packages)

• What this is: Apply OS/package updates and reboot if needed.
• Why it matters: Security fixes and stability; missing patches can leave vulnerabilities.
• When it’s due: Every 30 days (warn inside 7 days).
• Checklist:
  • Open Runbook Prompts (West/East patch + reboot)
  • SSH to each server (west-01, east-01).
  • Run apt update && apt upgrade (or the host’s package manager).
  • If prompted, schedule/perform reboot during low traffic.
  • After reboot, verify key sites load (LRWD, traveltechus).
• Confirm success: No pending critical updates; dashboard server tile shows services OK and no reboot required.
• Troubleshooting:
  • If a package is held/broken: check /var/log/apt, consider apt --fix-broken install.
  • If services fail after update: check systemctl status <service>, review logs, restart.

Quarterly restore test (snapshot/backup)

• What this is: Restore a backup/snapshot to staging and ensure it works.
• Why it matters: Proves backups are usable; avoids surprises during real incidents.
• When it’s due: Every 90 days (warn inside 7 days).
• Checklist:
  • Pick the latest HST-dated website backup for a key site (e.g., LRWD or traveltechus).
  • Restore to a staging environment or new VM.
  • Update hosts/DB config as needed for staging.
  • Load the site and run a basic sanity check (homepage, login if applicable).
  • Record time-to-restore and any issues.
• Confirm success: Restored site loads in staging, data intact, and you documented restore time.
• Troubleshooting:
  • Backup won’t import: check archive integrity, try a different snapshot.
  • Permissions/config issues: fix file ownership, update env/DB strings for staging.

MCI site collaboration

Purpose: Karen works locally with Codex and feature branches; Nick deploys when ready.

Daily flow (Karen)

• Sync and start dev: git pull, npm ci (if needed), npm run dev, open http://localhost:3000.
• Edit locally with Codex and refresh the browser to validate changes.
• Push a branch and tell Nick the branch name + what changed.

Deploy flow (Nick)

• Review/merge Karen's branch into main.
• Deploy to server preview and verify the site.

Quarterly failover drill (LB)

• What this is: Practice failing LRWD traffic from West to East and back.
• Why it matters: Ensures the load balancer + origins really fail over when needed.
• When it’s due: Every 90 days (warn inside 7 days).
• Checklist:
  • During a quiet window, drain/disable West pool in Cloudflare LB.
  • Confirm traffic serves from East (dashboard LB tile should show Active: lrwd-east).
  • Run basic site checks from East (LRWD health page, key pages).
  • Re-enable West and confirm Active returns to lrwd-west.
  • Document any issues or slow steps.
• Confirm success: LB shows primary healthy after re-enable; site loads fine from both regions.
• Troubleshooting:
  • If LB won’t fail over: check origin health, SSL, and DNS settings in Cloudflare.
  • If East is slow/down: inspect east-01 services (nginx/php/mysql/redis) and logs.