The Two Axes of Scaling

When one server runs out of capacity, you have two choices.

Vertical scaling (scale up)

Replace the box with a bigger box. More CPU cores, more RAM, faster disks, faster NIC. The application and operating system do not change; the same process runs on a beefier machine.

---------- Vertical scaling ----------
  before:  [ 4 vCPU,  16 GB RAM ]   handles 1K req/sec
  after:   [ 32 vCPU, 128 GB RAM ]  handles 8K req/sec

Horizontal scaling (scale out)

Keep the boxes the same; add more of them. A load balancer distributes requests across the fleet.

---------- Horizontal scaling ----------
  before:  [ load balancer ] -> [ server ]                 1K req/sec
  after:   [ load balancer ] -> [ server ] [ server ] [ server ] [ server ]  4K req/sec

The Comparison Table

The table makes vertical sound bad. It is not - it is the right answer for plenty of workloads, especially early ones. The question is which one you should default to.

Stateless vs Stateful (the real fork)

The scaling decision is downstream of one design choice: does your service hold state between requests?

Stateless service

Every request is self-contained. The server holds no memory of previous requests; any node in the fleet can handle any request equally.

Examples: REST API servers, image transcoders, render workers, function-as-a-service handlers.

Stateless services scale horizontally trivially. Add a node, the load balancer starts sending it traffic, no further coordination needed.

---------- Stateless: any node handles any request ----------
  request -> [ load balancer ] -> [ node 1 ] [ node 2 ] [ node 3 ]
              all nodes are interchangeable

Stateful service

The server holds data that persists between requests: a database row, a session, a partial computation, an open connection.

Examples: databases, caches, message queues, WebSocket servers holding live connections, in-memory game state.

Stateful services cannot just be cloned. The new node has no data; the existing nodes must hand off or replicate state. There are three ways out:

1. Externalize the state so the service itself becomes stateless (move sessions to Redis, cache to Memcached, files to S3).
2. Replicate the state so every node has a copy (works up to a point; replication overhead grows with N).
3. Shard the state so each node owns a slice (the only path that scales, at the cost of complexity).

The 'make it stateless' trick

The single most common scaling unlock in modern web architecture is to push state out of the request-handling layer.

Before: web server stores user sessions in process memory. The user must come back to the same node (sticky sessions) or re-login. Adding a node helps with new traffic but cannot help that user's existing session.

After: web server reads the session from Redis on every request. Any node can handle the user's next request. Add 100 nodes; throughput scales 100x.

Pseudocode for moving session state out

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

// Stateless handler: pulls session from Redis on every request.
app.get('/me', async (req, res) => {
    const sessionId = req.cookies.sid;
    const sessionRaw = await redis.get(`session:${sessionId}`);
    if (!sessionRaw) return res.status(401).end();
    const session = JSON.parse(sessionRaw);
    res.json({ userId: session.userId, role: session.role });
});

Notice: the handler now needs zero in-process state. Any number of API nodes can serve any user.

Where the Bottleneck Moves

Adding capacity does not eliminate bottlenecks; it shifts them. Knowing where the new bottleneck appears is what makes a senior engineer's scaling answer feel earned.

The last row is the important one: horizontal scaling does not help with lock contention. If 10K requests per second all want to UPDATE the same inventory row, adding more API servers makes the contention worse, not better. The fix is at a different layer (sharding the row, queuing the writes, or using optimistic concurrency).

When Vertical Scaling Is Still the Right Answer

Despite the case for horizontal, vertical wins in three situations.

1. Small workloads

A service handling 100 req/sec on a 4 vCPU box does not need a load balancer, service discovery, or rolling deploys. You will pay 10x the operational complexity for nothing. Vertical scaling - or not scaling at all - is the right call until you have evidence you need horizontal.

2. Single-writer stateful systems

A Postgres primary, a Kafka controller, a Zookeeper ensemble leader, a Redis Cluster master - these are inherently single-process for correctness. You scale them vertically until you can shard them. A 32-core, 256 GB Postgres box can serve a remarkable amount of traffic before sharding is worth the complexity.

3. Latency-sensitive in-process operations

When request latency is dominated by a single CPU-bound operation (ML inference, video transcoding, a complex SQL query), a faster CPU helps directly. Adding a second box does not make the operation faster; it just lets you do more of them in parallel. If you need to do the same one faster, vertical wins.

A Cost Model You Can Quote

On AWS today (rough memory), an m6i.large (2 vCPU, 8 GB) costs about $0.10/hour. An m6i.32xlarge (128 vCPU, 512 GB) costs about $6.10/hour. The big box is 64x the cores at 61x the cost - close to linear pricing.

But the operational cost of horizontal scaling is real: load balancer ($25 to $100/month), monitoring per node, more deployment surface, more failure modes. For workloads under ~$500/month in compute, vertical is often cheaper end to end. Above that, horizontal usually wins.

This matters in interviews: when you say 'I would scale horizontally', be ready for 'why not just buy a bigger box?'. The answer is some version of 'because at our load, the operational cost is dominated by raw compute, and horizontal gives me availability and zero-downtime deploys for free'.

A Mental Algorithm for the Interview

1. Is the service stateless? If yes, default to horizontal. If no, can you push the state out (Redis, S3, DB)? If yes, do that and now it is stateless.
2. What is the request profile? Read-heavy stateful: replicate (read replicas). Write-heavy stateful: shard. Single-writer with low write QPS: vertical until painful.
3. Where is the next bottleneck? Adding 10 more API servers might overwhelm the database. Plan the next layer's scaling strategy in the same answer.
4. What about availability? Horizontal gives N+1 redundancy for free. Vertical leaves you with one giant single point of failure.
5. What about cost? For tiny workloads, vertical. For anything serious, horizontal.

Walk this algorithm out loud in interviews and you will sound like you have done it before, because you have.