Motivation

The traditional model of running a service is: you pick an instance size, you start the process, you keep it running 24/7, you scale it manually or with auto-scaling rules, and you pay for every hour the instance exists whether it is doing work or not.

For a service with steady traffic that fully utilizes its CPU and memory, this is fine. For most workloads, it is wasteful:

• A nightly batch job: 10 minutes of work; 23 hours 50 minutes of idle paid compute.
• An image-resize endpoint that gets 5 requests per minute on average and 200 per minute during a viral moment: dramatically over- or under-provisioned almost all the time.
• A webhook receiver: bursty by definition; sometimes zero traffic for hours.
• A small internal API: maybe 1000 requests per day; a full instance is enormous overkill.

Serverless inverts the model: the cloud provider keeps the runtime warm somewhere, you give them code that runs per event, and you pay only for the milliseconds your code actually executed. Idle costs nothing. Auto-scaling is automatic and goes to zero. Operations (patching, capacity planning, instance failure recovery) are no longer your problem.

The price you pay: cold starts, vendor-specific runtimes, hard execution limits, painful state management, observability that is harder than it should be, and a cost model that flips on you at high steady throughput.

Knowing both the wins and the costs is what makes the architectural choice 'serverless or not' a senior-level decision rather than a 'because everyone does it' default.

Deep Dive

What 'serverless' actually means

The term spans several distinct services. Pinning down exactly what you mean is half of any interview answer.

The common thread: no instances to manage, scale to zero when idle, billed by usage. This lesson focuses primarily on FaaS because it is the prototype, but the pattern generalizes.

How a FaaS request actually executes

When a function is triggered (HTTP, event, schedule, queue message), the provider needs an executable environment. There are three lifecycle states.

+--- Init (cold start) ----+   +--- Invoke (warm) ----+   +--- Idle ----+
| 1) Find/start a sandbox  |   | reuse same sandbox   |   | sandbox     |
| 2) Download code/layers  |-->| just call the handler|-->| kept warm   |
| 3) Start runtime         |   | (1-10 ms overhead)   |   | for ~5-15 m |
| 4) Run init code         |   |                      |   |             |
| 5) Call handler          |   |                      |   |             |
+--------------------------+   +----------------------+   +-------------+
      typically 50ms - 2s          fast                       provider tears
      (much worse for JVM/        path                        down silently
      .NET, much better for
      Go / Rust / Cloudflare
      isolates)

Cold start: the first request to a new sandbox pays for sandbox creation, code download, runtime startup, and any init code (e.g., loading models, opening connections). Cold-start times in 2026 typically range from ~50 ms (Cloudflare Workers, AWS Lambda SnapStart) to ~2 s (large JVM or .NET Lambdas with heavy init).

Warm invocation: subsequent requests to the same sandbox skip init; only the handler runs. Per-invocation overhead is in single-digit milliseconds.

Idle: after some time without traffic (typically 5-15 minutes), the provider tears down the sandbox. The next request pays a cold start.

Why does this matter? Because cold starts dominate the tail of latency for low-traffic functions. A function that gets one request per minute pays a cold start about every minute; its p99 will be the cold-start time, not the warm-path time.

Concurrency model

A FaaS function processes one request per sandbox at a time (the standard model in AWS Lambda, Cloud Functions, Azure Functions). If 100 requests arrive simultaneously, the provider spins up 100 sandboxes. This is wonderful for scaling but it has consequences:

• Connection storms: every cold sandbox opens its own database connection. 1000 concurrent invocations = 1000 DB connections. RDS connection limits get hit fast.
• Burst limits: every provider has a per-account concurrency cap (e.g., AWS Lambda's 1000 default per region). Hit it and requests get throttled.
• No internal state: a counter you increment in memory dies with the sandbox; a cache you build in memory benefits one sandbox only.
• Per-request CPU and memory: AWS Lambda allocates CPU proportionally to memory. A 128 MB function gets a small fraction of a core; a 1769 MB function gets a full vCPU.

Cloudflare Workers and similar 'isolate'-based runtimes break the one-request-per-sandbox rule (multiple requests share a V8 isolate), trading some isolation for much lower per-request overhead and far better cold-start behavior.

Hard limits to respect

Every FaaS platform has rigid limits that constrain architecture:

These are not optional. Designing serverless without naming them is a junior signal.

State: where it lives

FaaS is fundamentally stateless compute. You cannot keep state across invocations. Every state need has to land somewhere external:

• Caching layer: Redis (ElastiCache / MemoryDB), DynamoDB, or in-memory only when warmth is acceptable.
• Database: DynamoDB and similar serverless databases pair naturally; relational DBs need a connection pooler (RDS Proxy) to survive concurrency storms.
• Workflow state: Step Functions or Durable Functions store the state of a multi-step workflow so individual steps can be stateless.
• Object storage: S3 / GCS for any large artifacts (uploads, intermediate files).

A realistic serverless architecture is one part code, three parts managed services.

Triggers: the event surface

The interesting thing about FaaS is that anything can be a trigger:

• HTTP request via API Gateway or function URL.
• Object created in S3.
• Message arrived in SQS, SNS, EventBridge, Kinesis.
• Row changed in DynamoDB stream.
• Scheduled by cron expression.
• IoT device event.
• Workflow step in Step Functions.

This trigger ecosystem is the real win. Wiring 'when a file lands in S3, transcode it, store the result, send a notification' becomes four small functions and three event subscriptions. No long-running orchestrator to babysit.

Implementation

A canonical serverless pipeline

A classic 'upload, process, notify' workflow looks like this:

user upload
      |
      v
  +-------------+  S3:ObjectCreated  +------------+ writes to +-----------+
  | S3 bucket   | -----------------> | Lambda A   | --------> | DynamoDB  |
  +-------------+                    | (process)  |           +-----------+
                                     +-----+------+
                                           |
                                           | publish event
                                           v
                                     +-------------+
                                     | EventBridge |
                                     +------+------+
                                            |
                                            v
                                     +-------------+    +----------------+
                                     | Lambda B    |--> | SES / SNS push |
                                     | (notify)    |    +----------------+
                                     +-------------+

Notable features:

• No servers to manage anywhere. Every box is managed.
• Failure isolation: if Lambda B fails, retried independently; if EventBridge degrades, both Lambdas can be invoked from a dead-letter queue.
• Scale: the pipeline auto-scales from 1 file per hour to 10,000 per minute without any code change.
• Cost at idle: zero. You pay only when files arrive.

Cold start mitigation

Four established techniques:

1. Provisioned concurrency (AWS Lambda) / min instances (Cloud Run, Cloud Functions): keep N sandboxes pre-warmed. Pay for them whether used or not. The classic 'serverless minus the scale-to-zero benefit' trade.
2. SnapStart (AWS Lambda Java, .NET, Python): take a snapshot of the initialized runtime and resume from snapshot instead of cold-booting. Often 90% reduction in cold-start time.
3. Smaller, faster runtimes: Go, Rust, and Node tend to have sub-100 ms cold starts; large JVM apps have multi-second cold starts. Choosing the runtime is choosing the cold-start floor.
4. Architecture: keep latency-sensitive paths off FaaS or put a fast cache in front of them.

Not a fix: 'pinging the function every minute to keep it warm'. This works for one sandbox. Under any concurrency, new sandboxes still cold-start.

The connection-storm problem

A naive serverless function that opens a Postgres connection per invocation works fine at low traffic. Under burst it opens hundreds or thousands of connections in seconds and the database falls over.

The standard fixes:

• Use a connection pooler outside the function (RDS Proxy, PgBouncer). The function sees a small pool; the pool multiplexes onto a small number of real DB connections.
• Use a serverless database (DynamoDB, Aurora Serverless v2, Firestore) that scales connections naturally.
• Use HTTP-based data APIs (Aurora Data API, Hasura, Supabase REST) that hide the connection layer.
• Keep connections in module scope (outside the handler) so warm invocations reuse them within a sandbox. This caps connections at 'max concurrency' rather than 'requests per second'.

Observability is harder

With serverless you get:

• Metrics: per-function invocation count, duration, errors, throttles, cold-start count. Easy.
• Logs: per-invocation logs to CloudWatch / Stackdriver. Easy but expensive to query at scale.
• Traces: distributed tracing across function chains needs explicit propagation (X-Ray, OpenTelemetry SDK). Harder.
• Profiling and debugging: no SSH to attach a debugger. You add log statements and redeploy. The feedback loop is slower than for a long-running service.

A real serverless platform investment includes structured logging, trace context propagation, and cost-aware log retention.

Cost model and the crossover point

Serverless cost is roughly:

cost = invocations * unit_price + GB-seconds * memory_price

AWS Lambda (representative 2026 pricing):

• $0.20 per million requests.
• $0.0000166667 per GB-second of compute.

Worked example: a 200 ms function with 512 MB of memory invoked 10 million times a day:

• Requests: 10M * 30 days = 300M -> ~$60/month for invocation cost.
• GB-seconds: 0.5 GB * 0.2 s * 300M = 30M GB-s -> ~$500/month for compute.
• Total: ~$560/month.

A t3.medium EC2 instance running the same workload as a long-running service can be had for ~$30/month. Even with 4-5 instances behind a load balancer for redundancy and burst, you are looking at ~$200/month all-in.

At steady high throughput, dedicated compute is dramatically cheaper. The break-even point is workload-dependent but usually lands around 'busy more than ~30% of the day'. Below that, serverless wins on cost AND operations.

When to Use

Strong fit for serverless

• Spiky / unpredictable traffic (webhook receivers, image processing, IoT ingestion).
• Event-driven glue code: 'when X happens, do Y' wiring between managed services.
• Low-traffic APIs and admin tools where idle cost dominates.
• Scheduled jobs: cron-style work that runs minutes per day.
• Stateless transformations: data validation, format conversion, enrichment.
• Edge compute: Cloudflare Workers / Lambda@Edge for personalization, A/B testing, auth at the edge.
• Prototypes and MVPs where shipping speed matters more than per-request cost.

Probably do not use serverless

• Steady high-throughput services where dedicated compute is cheaper and warmer.
• Latency-critical paths (sub-50ms p99 targets) where cold starts can blow the SLO.
• Long-running jobs beyond the platform's max execution time (15 min on Lambda).
• Stateful workloads that benefit from in-memory state, sticky sessions, or open connections.
• WebSocket / streaming connections at scale (the connection model fights the per-invocation billing).
• Heavy CPU or GPU work that exceeds FaaS resource limits.
• Vendor-portability requirements where being locked into provider-specific event services is unacceptable.

Compared to alternatives

Hybrid is the realistic answer

Most serious companies running serverless run a hybrid: long-running services for the steady core, FaaS for the spiky edges (webhooks, image processing, scheduled tasks). The interview answer 'serverless for these workloads, dedicated compute for these other workloads, here is my reason' is consistently stronger than 'all-in on either side'.

Case Studies

iRobot: serverless from day one

iRobot publicly described running their entire backend (millions of Roomba devices, billions of telemetry events) on AWS Lambda + DynamoDB + Kinesis with no traditional servers. Their team is famously small for the workload because they never built any of the infrastructure those services replace. This is one of the canonical 'serverless at scale' references.

Lesson: an event-driven IoT workload is almost the textbook fit for serverless, and the operational savings are enormous when the team is small relative to the workload.

Netflix: serverless for media pipelines

Netflix uses AWS Lambda extensively for video processing pipelines: each new master file triggers transcoding, validation, packaging, and publishing through a chain of functions. The streaming serving path itself remains on long-running services because of the QPS and latency profile, but the asynchronous media-processing work is a natural FaaS fit.

Lesson: a single company's architecture mixes serverless and dedicated compute deliberately; the boundary is set by per-workload economics and latency.

Coca-Cola vending machines

Coca-Cola described a serverless rewrite of their vending-machine payment system that cut backend operating cost dramatically by moving from always-on EC2 to per-transaction Lambda. The workload is naturally bursty (transactions during the day, near zero overnight) and matches the serverless cost model precisely.

Lesson: workloads with high idle ratios are where serverless wins on raw dollars, not just operations.

A typical cautionary tale: the always-busy API

Many engineering blogs describe teams porting a steady-throughput API (constant ~1000 RPS) to Lambda and watching the bill explode. The cost per million invocations multiplied by the constant traffic exceeded what a small autoscaling group of EC2s would cost by 5-10x. The teams either added provisioned concurrency (which mostly negates the serverless cost advantage) or migrated those endpoints back to ECS / Fargate.

Lesson: serverless is not a universal cost win. The break-even with dedicated compute lands in favor of dedicated when traffic is steady and high.

Cloudflare Workers: a different cold-start story

Cloudflare Workers run on V8 isolates rather than per-request containers. Cold starts are typically <5 ms because there is no container to spin up; the runtime stays warm and the function is just a new isolate inside it. This makes Workers viable for edge personalization on every page load, a workload that would be marginal on traditional Lambda due to cold starts.

Lesson: 'serverless cold start' is platform-dependent. Pick the runtime that matches the latency requirements; isolates are a different point on the trade-off curve from per-request containers.

Quick Review

• Serverless = scale to zero, pay per usage, ops handled by the provider. FaaS is the most visible flavor.
• Cold starts dominate the tail latency of low-traffic functions; warm invocations are fast.
• Concurrency model is one request per sandbox (except for isolate runtimes like Cloudflare Workers).
• Hard limits (15 min max runtime, 10 GB memory, 6 MB payload, account concurrency caps) drive architecture.
• State lives outside: in serverless DBs, caches, object storage, or workflow engines.
• Cost wins for spiky workloads; loses for steady high-throughput workloads.
• Hybrid (FaaS at the edges, dedicated compute for the steady core) is the realistic enterprise pattern.