The One-Hour VPS Incident Drill: A Runbook You Can Rehearse Monthly

The One-Hour VPS Incident Drill: A Runbook You Can Rehearse Monthly

If your team has never practiced an outage, your first real outage is your practice run. That is expensive.

This article gives you a 60-minute drill that works for small teams with limited time. No theater, no fake enterprise process, just direct operational training.

Goal of the drill

You are not trying to prove technical genius. You are testing whether your team can:

• detect impact quickly
• coordinate clearly
• recover service safely
• capture lessons before they are forgotten

Role cards (assign in 5 minutes)

Use four roles even in a tiny team:

1. Incident lead: owns decisions and timeline.
2. Operator: executes commands and mitigation steps.
3. Communications owner: posts status updates.
4. Scribe: logs actions, timestamps, and open questions.

One person can hold two roles, but never combine incident lead and scribe.

Drill scenario (starter)

Use this baseline:

• Primary VPS app becomes intermittently unavailable.
• Latency spikes, then 502 errors start appearing.
• Public status page still says all systems normal.

This scenario is realistic enough for web apps and API workloads.

60-minute timeline

00:00-10:00: Detection and triage

• Trigger alert and give only minimal context.
• Team identifies user impact and confirms reproduction.
• Incident lead declares severity and opens incident channel.

Pass criteria: incident is formally declared in less than 10 minutes.

10:00-25:00: Stabilization

• Operator checks host health: CPU, memory, disk saturation, network errors.
• Team chooses one mitigation path (scale up, restart service, disable heavy background job, or route traffic to fallback).
• Communications owner posts a customer-safe status note.

Pass criteria: one mitigation is executed and communication is published.

25:00-40:00: Root cause narrowing

• Validate logs around first symptom timestamp.
• Correlate app errors with infrastructure signals.
• Reject at least one false hypothesis explicitly.

Pass criteria: top two probable causes are documented, not guessed.

40:00-50:00: Recovery confirmation

• Confirm error rate and latency return to acceptable range.
• Verify key user journeys, not only health endpoints.
• Define guardrails to prevent immediate recurrence.

Pass criteria: recovery is confirmed with data, not assumptions.

50:00-60:00: Debrief

• Scribe reads timeline from first alert to recovery.
• Team lists three things that worked and three that did not.
• Assign exactly two improvements with owners and deadlines.

Pass criteria: action items are specific and scheduled.

Scoring model (keep it simple)

Score each category from 1 to 5:

• Clarity of ownership
• Time to first customer-facing update
• Safety of mitigation
• Evidence quality in root cause analysis
• Actionability of follow-up items

A team averaging below 3 should repeat the same scenario next month before moving to harder drills.

Mistakes to avoid during drills

1. Turning drill into a blame session.
2. Skipping communications because “this is only internal.”
3. Running perfect scripts that do not resemble production reality.
4. Ending without owners on improvement items.

The drill fails if learning does not survive past the meeting.

Monthly cadence suggestion

Alternate scenario categories:

• month 1: resource exhaustion (CPU, disk, memory)
• month 2: dependency failure (DB, cache, DNS)
• month 3: deployment regression

Rotate incident lead role every month to spread operational confidence.

Closing note

A one-hour drill is cheap compared with one hour of unpracticed downtime. The teams that recover fast are usually not “smarter”; they are simply more rehearsed.