CONFIDENTIAL · TEAM EVIDENCE PACK

Who Killed Checkout?

ChaiCart Incident INC-0001 · Launch Day

09:00 — ChaiCart launched in 5 cities. A push notification went to 2.1 million users.

09:03 — Checkout success rate crashed from 99% to 38%. Customers see “Payment stuck on spinner…” for 30 seconds, then an error.

09:10 — The CEO is drafting an angry tweet. You have 25 minutes to find the killer before it's posted.

Your evidence

Exhibit	Contents
A	Metrics dashboard (08:45 – 09:15)
B	Logs from all services
C	Three distributed traces
D	Change log & tickets
E	Witness statements
F	Bonus: a code snippet (for the brave)
—	Accusation Form (hand this in!)

ChaiCart's checkout path

app → checkout-service → cart-service
↳ payment-service → orders-db (PostgreSQL) + PayFast gateway (external)
Also running: redis cache (menus & carts), log-server (collects logs)

The suspects

🗄️

Postgres Pete
orders database

“I was just sitting there. Nobody even talked to me.”

🚀

Deploy Dave
payment-service v2.3.1

“Tiny config tweak. FinOps approved it!”

💳

PayFast Priya
external payment gateway

“My status page is green.”

💾

Disk-Full Dinesh
log-server disk at 97%

“I've been this full for two days. Nobody cares.”

⚡

Redis Rani
cache

“Hit ratio 94%, same as always.”

📈

Traffic Tara
launch surge, 3×

“You planned for 5×. Don't blame me.”

Exhibit A — Metrics Dashboard

All charts: 08:45 → 09:15, one point per minute. Dashed line = 09:00 launch push notification.

Exhibit B — Logs

Collected from all services. Sorted by time. Not everything here is relevant!

[2 days ago]
18:02:11 INFO  [payment-service] Starting payment-service v2.3.0 (build 51d0e2c)
18:02:12 INFO  [payment-service] HikariPool-1 - configuration: maximumPoolSize=50 connectionTimeout=30000
18:40:03 WARN  [log-server] Disk usage at 96% on /var/log (threshold 95%)

[Launch day]
08:45:00 INFO  [redis] Keyspace hits=94.2% evicted_keys=0
08:50:14 INFO  [checkout-service] POST /api/checkout 200 (312 ms) trace=a1f3…
08:55:40 WARN  [log-server] Disk usage at 97% on /var/log (threshold 95%)
08:58:02 INFO  [payment-service] Starting payment-service v2.3.1 (build 7f3c9a1)
08:58:03 INFO  [payment-service] HikariPool-1 - configuration: maximumPoolSize=5 connectionTimeout=30000
08:58:05 INFO  [payment-service] HikariPool-1 - Start completed.
08:58:30 INFO  [orders-db] connection closed: user=payment_svc (x45)
09:00:00 INFO  [notification-service] Campaign LAUNCH sent to 2,104,332 devices
09:00:41 INFO  [checkout-service] POST /api/checkout 200 (298 ms)
09:01:52 WARN  [payment-service] HikariPool-1 - Pool stats (total=5, active=5, idle=0, waiting=31)
09:02:13 WARN  [payment-service] HikariPool-1 - Pool stats (total=5, active=5, idle=0, waiting=87)
09:02:43 ERROR [payment-service] HikariPool-1 - Connection is not available, request timed out after 30000ms.
09:02:43 ERROR [checkout-service] POST /api/checkout failed: 504 Gateway Timeout from payment-service (30012 ms)
09:02:44 INFO  [redis] Keyspace hits=94.1% evicted_keys=0
09:02:55 WARN  [log-server] Disk usage at 97% on /var/log (threshold 95%)
09:03:10 INFO  [orders-db] checkpoint complete: wrote 312 buffers (0.4%); active connections=5
09:03:12 ERROR [payment-service] HikariPool-1 - Connection is not available, request timed out after 30000ms.
09:03:14 INFO  [payfast-client] Gateway health check OK (latency 204 ms)
09:03:30 WARN  [frontend] Client error rate spike: "Payment stuck on spinner" x 1,284
09:03:41 INFO  [checkout-service] POST /api/checkout 200 (18,902 ms)
09:04:02 ERROR [payment-service] HikariPool-1 - Connection is not available, request timed out after 30000ms.
09:04:05 INFO  [orders-db] LOG: duration: 11.8 ms  statement: INSERT INTO payments …
09:04:20 INFO  [cart-service] GET /api/cart 200 (17 ms)
09:05:00 WARN  [payment-service] HikariPool-1 - Pool stats (total=5, active=5, idle=0, waiting=214)

Exhibit C — Distributed Traces

Each bar is a span. Bar length is proportional to duration within that trace. Striped = waiting.

Exhibit D — Change Log & Tickets

When	What	Details
2 days ago 18:40	Alert	OPS-311 “log-server disk > 95%” — acknowledged, “clean old logs after launch”.
Yesterday 18:00	Deploy	frontend v5.0 — new launch banner & animations. Load tested: yes.
Today 08:30	Config	redis — no change, health check only.
Today 08:58	Deploy	payment-service v2.3.1 — FINOPS-482 “Right-size DB tier: reduce connection pool to lower DB cost”. Reviewer: cost-bot (approved). Load test: skipped — launch day rush. Rollout: 100% at once.
Today 09:00	Marketing	Launch push notification to 2.1M users. Expected traffic: up to 5× normal.

Exhibit E — Witness Statements

Meera, Support Lead: “From about 09:03 we got hundreds of tickets: ‘Payment stuck on spinner, then fails.’ Nobody complained about menus or carts being slow.”

Arjun, DBA: “The database is bored! CPU under 30%. Actually I saw fewer connections than usual this morning — I thought it was a quiet day.”

Kabir, On-call SRE: “My pager went off for disk on the log server… but that alert has been firing for two days, so I muted it.”

Tanya, Head of Marketing: “We sent the push at 09:00 sharp. We told engineering to expect 5×. We only got 3×! This is not on us.”

Rohan, FinOps Analyst: “Great news — we're on track to save 22% on the database bill this month. Engineering made some tuning changes for us.”

PayFast Account Manager: “Our latency is completely normal. Honestly, we expected many more transactions from your launch.”

Exhibit F — Bonus: Code Snippet

From payment-service. For teams who want the extra credit (+25): what makes the problem worse?

@Transactional   // opens a DB connection for the whole method
public PaymentResult pay(Order order) {
    paymentsRepo.insertPending(order);            // ~12 ms on the DB
    GatewayResponse r = payFast.charge(order);    // ~205 ms external HTTP call
    paymentsRepo.markResult(order, r.status());   // ~5 ms on the DB
    return PaymentResult.from(r);
}

Accusation Form

Team name: ______________________ Time handed in: ________

The killer (who)
The weapon (how exactly?)
Accomplice (if any)
Evidence – a metric
Evidence – a log line
Evidence – a trace
Why the others are innocent
Immediate fix
How to prevent it next time

FACILITATOR ONLY — DO NOT PUT IN ENVELOPES

Solution: Who Killed Checkout?

Killer	Deploy Dave — payment-service v2.3.1 (08:58) reduced `maximumPoolSize` from 50 to 5 (FINOPS-482). No load test and 100% rollout.
Weapon	DB connection pool exhaustion. With 3× traffic, only 5 connections exist. Each payment holds a connection for ~220 ms (including the PayFast call, see Exhibit F), so requests queue. After 30 s, `Connection is not available, request timed out after 30000ms` → checkout 504.
Accomplice	Traffic Tara — the 09:00 surge was the trigger. With pre-launch traffic, 5 connections were barely enough, so nothing failed between 08:58 and 09:01.
Key evidence	Metric: DB active connections drop 42 → 5 at 08:59; DB CPU flat ~28%; payment p95 jumps to 30,000 ms; PayFast calls cap at ~450/min while traffic is ~1,200/min (throughput ceiling = saturation). Logs: `maximumPoolSize=5` (was 50 two days ago); `Pool stats … idle=0, waiting=87`; the timeout error. Traces: failed trace — 30,000 ms in `HikariPool.getConnection`, no DB or PayFast span; slow success — 18.4 s waiting, then normal DB & PayFast times.
Innocent	Postgres Pete: low CPU, fast queries (11.8 ms) — starved, not overloaded. PayFast Priya: latency flat ~205 ms; failed traces never reach her. Redis Rani: hit ratio flat 94%, carts fast (17 ms). Disk-Full Dinesh: 97% for two days — a real issue, but an unrelated red herring (alert fatigue!).
Fix	Roll back to v2.3.0 (or set pool back to 50) — recovery within minutes.
Prevention	Load/performance test in CI · canary rollout with automatic rollback on SLO breach · alert on pool saturation (waiting > 0) · deployment events on dashboards · cost changes need reliability review · don't hold DB transactions across external calls · fix noisy alerts to avoid alert fatigue.
Scoring	Killer +200 · Weapon +100 · Accomplice +50 · Fix + prevention +50 · Exhibit F insight +25 · First correct team +100 · Wrong killer −50