Fault Tolerance
fault-tolerance
Practice Problems
Bounded Retry Policy
Decide whether to retry a failed provider call: stop for unsafe replay, a permanent failure, used-up attempts, or a deadline that cannot fit the wait, and otherwise return a capped, jittered backoff delay that honors retry-after.
System Design
Leader Election & Consensus (Raft, Paxos)
Leader election is how a distributed cluster picks one node to be in charge so the others can stop arguing. This lesson covers the consensus problem (FLP impossibility), Paxos in concept, Raft in detail (leader election + log replication + safety), the role of quorum, and the operational pitfalls of split brain and network partitions. We also tour the systems that ship Raft or Paxos in production: etcd, ZooKeeper, Consul, CockroachDB, MongoDB, Spanner. By the end you can explain why every modern distributed database has a consensus protocol at its core, and you can sketch Raft on a whiteboard.
Fault Tolerance, Redundancy & Failover
Fault tolerance is the property that lets a system keep working when components fail - and at any reasonable scale, components are always failing. This lesson covers the building blocks: redundancy (active-active, active-passive), failure detection (health checks, heartbeats), failover (automatic, manual), and the patterns that make systems gracefully degrade instead of catastrophically crash (circuit breakers, retries with backoff, bulkheads, timeouts). We finish with the operational disciplines that turn architecture into reality: chaos engineering, runbooks, blast-radius analysis, and disaster recovery (RTO/RPO). By the end you can design a system that survives the failure modes interviewers love to throw at you.
AI Engineering
Design an agent with idempotent side effects
Design an agent that closes a ticket and sends one approved closure email safely across duplicate jobs, crashes, timeouts, and concurrent workers, and say exactly what the design can and cannot guarantee.
Implement a bounded retry policy
Implement a pure, deterministic retry policy with attempt limits, capped jittered backoff, retry-after, and a deadline check, then fit it into an executor that handles cancellation, partial streams, and uncertain writes.
Design SLOs, rollout, and cost controls
Design reliability controls for a multi-tenant AI assistant: SLOs and error budgets, quota-aware admission, fallback and cost reservations, staged rollout with a canary decision, rollback, and recovery.