Incident Debrief Questions They Asked Me
A 4-question set drawn from the debrief portion of an SRE-flavored loop. Every behavioral prompt about an on-call story got paired with a design follow-up the interviewer used to stress-test the takeaway.
By @lilyadeyemi
December 18, 2025
·
Updated August 12, 2026
799 views
25
Rate
They opened with: "Walk me through the worst on-call page you owned." After the story they asked: "Now show me the retry-with-jitter you would have added to that downstream call." Trace the backoff I sketched.
Post-incident
I traced one call on the board:
attempts = [retry_delay(0), retry_delay(1), retry_delay(2), retry_delay(3)]
# attempts roughly = [random in (0, 0.1), (0, 0.2), (0, 0.4), (0, 0.8)] seconds
# uniform jitter spreads retries instead of stampeding the downstream at fixed intervals.import random
import time
def retry_delay(attempt: int, base: float = 0.1, cap: float = 5.0) -> float:
sleep = min(cap, base * (2 ** attempt))
return random.uniform(0, sleep)
def call_with_retry(fn, max_attempts: int = 5):
last = None
for attempt in range(max_attempts):
try:
return fn()
except Exception as err:
last = err
time.sleep(retry_delay(attempt))
raise lastThe debrief question: "What did you change in the runbook after that incident?" Follow-up: "Show me the structured log line you added so the next on-call would not have to grep three services." Walk through the schema I drew.
Post-incident
I drew the log entry on the board:
log_event(trace_id="abc-123", service="orders", level="ERROR",
event="payment_decline", customer_id="c-42",
downstream="stripe", status_code=429)
# Single line, machine-parseable, contains enough to pivot in Datadog without opening three dashboards.import json
import time
def log_event(**fields) -> None:
record = {
"ts": time.time(),
"level": fields.pop("level", "INFO"),
**fields,
}
print(json.dumps(record, sort_keys=True))
class StructuredLogger:
def __init__(self, service: str, trace_id: str):
self.service = service
self.trace_id = trace_id
def error(self, event: str, **kw) -> None:
log_event(service=self.service, trace_id=self.trace_id,
level="ERROR", event=event, **kw)Asked: "What detection signal would have caught the regression earlier?" Then: "Show me the alert rule you would have added on that metric." Walk through the rate-of-change check I drew.
Post-incident
I drew the check on the board:
breach = alert_if_rate_drops(success_rate_per_minute, window=5, threshold=0.2)
# If the rolling 5-minute success rate falls more than 20% relative to the previous 5 minutes, page.from collections import deque
class RateChangeAlert:
def __init__(self, window: int = 5, threshold: float = 0.2):
self.window = window
self.threshold = threshold
self.recent = deque(maxlen=window * 2)
def observe(self, value: float) -> bool:
self.recent.append(value)
if len(self.recent) < self.window * 2:
return False
previous = list(self.recent)[: self.window]
current = list(self.recent)[self.window :]
prev_avg = sum(previous) / self.window
curr_avg = sum(current) / self.window
if prev_avg == 0:
return False
drop = (prev_avg - curr_avg) / prev_avg
return drop >= self.thresholdFinal ask: "What would you ship before next on-call rotation?" Follow-up: "Sketch the circuit breaker you would put in front of that flaky dependency." Walk through the three-state machine I drew.
Post-incident
I drew the breaker state transitions:
breaker = CircuitBreaker(failure_threshold=5, recovery_seconds=30)
breaker.call(downstream) # closed -> failing requests trip after 5 -> open
breaker.call(downstream) # open -> raises immediately, no downstream hit
# 30s later: half-open -> one trial call, success closes the breaker.import time
class BreakerOpen(Exception): pass
class CircuitBreaker:
def __init__(self, failure_threshold: int = 5, recovery_seconds: float = 30.0):
self.failure_threshold = failure_threshold
self.recovery_seconds = recovery_seconds
self.failures = 0
self.state = "closed"
self.opened_at = 0.0
def call(self, fn):
if self.state == "open":
if time.monotonic() - self.opened_at >= self.recovery_seconds:
self.state = "half-open"
else:
raise BreakerOpen()
try:
result = fn()
self.failures = 0
self.state = "closed"
return result
except Exception:
self.failures += 1
if self.failures >= self.failure_threshold or self.state == "half-open":
self.state = "open"
self.opened_at = time.monotonic()
raise