The One-Hour VPS Incident Drill: A Runbook You Can Rehearse Monthly

The One-Hour VPS Incident Drill: A Runbook You Can Rehearse Monthly

If your team has never practiced an outage, your first real outage is your practice run. That is expensive.

This article gives you a 60-minute drill that works for small teams with limited time. No theater, no fake enterprise process, just direct operational training.

Safety note (keep it non-destructive)

• Default to a tabletop drill: do not intentionally break production.
• Simulate symptoms with dashboards, logs, or staging traffic.
• If you practice real mitigations, define stop conditions and a rollback path first.

Goal of the drill

You are not trying to prove technical genius. You are testing whether your team can:

• detect impact quickly
• coordinate clearly
• recover service safely
• capture lessons before they are forgotten

Role cards (assign in 5 minutes)

Use four roles even in a tiny team:

1. Incident lead: owns decisions and timeline.
2. Operator: executes commands and mitigation steps.
3. Communications owner: posts status updates.
4. Scribe: logs actions, timestamps, and open questions.

One person can hold two roles, but never combine incident lead and scribe.

Prep checklist (2 minutes)

Before you start the timer, ensure:

• a shared dashboard is open (latency + error rate + host health)
• the incident channel and a single timeline doc exist
• the team agrees on a stop condition and rollback path

Drill scenario (starter)

Use this baseline:

• Primary VPS app becomes intermittently unavailable.
• Latency spikes, then 502 errors start appearing.
• Public status page still says all systems are normal.

This scenario is realistic enough for web apps and API workloads.

60-minute timeline

00:00-10:00: Detection and triage

• Trigger alert and give only minimal context.
• Team identifies user impact and confirms reproduction.
• Incident lead declares severity and opens incident channel.

Pass criteria: incident is formally declared in less than 10 minutes.

10:00-25:00: Stabilization

• Operator checks host health: CPU, memory, disk saturation, network errors.
• Team chooses one mitigation path (scale up, restart service, disable heavy background job, or route traffic to fallback).
• Communications owner posts a customer-safe status note.

Pass criteria: one mitigation is executed and communication is published.

25:00-40:00: Root cause narrowing

• Validate logs around first symptom timestamp.
• Correlate app errors with infrastructure signals.
• Reject at least one false hypothesis explicitly.

Pass criteria: top two probable causes are documented, not guessed.

40:00-50:00: Recovery confirmation

• Confirm error rate and latency return to an acceptable range.
• Verify key user journeys, not only health endpoints.
• Define guardrails to prevent immediate recurrence.

Pass criteria: recovery is confirmed with data, not assumptions.

50:00-60:00: Debrief

• Scribe reads timeline from first alert to recovery.
• Team lists three things that worked and three that did not.
• Assign exactly two improvements with owners and deadlines.

Pass criteria: action items are specific and scheduled.

Customer status note template (copy/paste)

Keep it short and avoid guessing root cause:

We are investigating reports of [symptom] affecting [service/region]. Mitigation is in progress and we are monitoring impact. Next update in [15 minutes] (at [time]).

Scoring model (keep it simple)

Score each category from 1 to 5:

• Clarity of ownership
• Time to first customer-facing update
• Safety of mitigation
• Evidence quality in root cause analysis
• Actionability of follow-up items

A team averaging below 3 should repeat the same scenario next month before moving to harder drills.

Mistakes to avoid during drills

1. Turning drill into a blame session.
2. Skipping communications because “this is only internal.”
3. Running perfect scripts that do not resemble production reality.
4. Ending without owners on improvement items.

The drill fails if learning does not survive past the meeting.

Monthly cadence suggestion

Alternate scenario categories:

• month 1: resource exhaustion (CPU, disk, memory)
• month 2: dependency failure (DB, cache, DNS)
• month 3: deployment regression

Rotate incident lead role every month to spread operational confidence.

Closing note

A one-hour drill is cheap compared with one hour of unpracticed downtime. The teams that recover fast are usually not “smarter”; they are simply more rehearsed.

Reference

• NIST incident handling guide (SP 800-61r2): csrc.nist.gov
• Incident response plan overview: PagerDuty