Card payments failing, 3 October

What happened, and how long did it take anyone to notice?

Errors climbed seven minutes after the 2.14 rollout, but the page fired 22 minutes later because the alert waits for a fixed 5% to hold. From the page, rollback took 28 minutes.

What it shows

11 of 11 file citations checked against the repository

The diagram as text

Events, in time order

03 Oct 09:12 to 06 Oct 14:00 (3d 4h).

  1. 03 Oct 09:12 v2.14 rolled out (Deploys) (oncall-notes.md:5 ✓)
  2. 03 Oct 09:19 Card errors pass 5% (Monitoring) (oncall-notes.md:6 ✓)
  3. 03 Oct 09:41 Pager fires (Monitoring) (oncall-notes.md:7 ✓)
  4. 03 Oct 09:44 Ana acknowledges (On-call) (oncall-notes.md:8 ✓)
  5. 03 Oct 09:51 Rollout named as the cause (On-call) (oncall-notes.md:9 ✓)
  6. 03 Oct 09:58 Rollback to v2.13 (Deploys) (oncall-notes.md:10-11 ✓)
  7. 03 Oct 10:11 Errors back under 0.5% (Monitoring) (oncall-notes.md:12 ✓)
  8. 03 Oct 10:30 Status page: resolved (On-call) (oncall-notes.md:13 ✓)
  9. 06 Oct 14:00 Postmortem (On-call) (oncall-notes.md:14 ✓)

Notes

● event◆ milestone▬ lasted from … to// a quiet stretch, shortened
DeploysMonitoringOn-call03 Oct 09:2009:3009:4009:5010:0010:1010:2010:3006 Oct 14:003d 3h latercauseddetected 22 min later03 Oct 09:12 — v2.14 rolled outv2.14 rolled out03 Oct 09:1203 Oct 09:19 — Card errors pass 5%Card errors pass 5%03 Oct 09:1903 Oct 09:41 — Pager firesPager fires03 Oct 09:4103 Oct 09:44 — Ana acknowledgesAna acknowledges03 Oct 09:4403 Oct 09:51 — Rollout named as the causeRollout named as the cause03 Oct 09:5103 Oct 09:58 — Rollback to v2.13Rollback to v2.1303 Oct 09:5803 Oct 10:11 — Errors back under 0.5%Errors back under 0.5%03 Oct 10:1103 Oct 10:30 — Status page: resolvedStatus page: resolved03 Oct 10:3006 Oct 14:00 — PostmortemPostmortem06 Oct 14:00