Networking and edge routing
One request's journey: DNS, anycast, CDNs, load balancers, proxies, API gateways, and real-time connections.
On this page
Before a request reaches any code you wrote, it has been resolved, routed, cached, balanced and proxied by a chain of machines that exist only to get it to the right place quickly. This article follows one request from a browser to an application server and stops at each hop to explain what it does and why it is there.
The map
Read this first when short on time. Every branch is a section below.
The journey in one picture
A request crosses roughly six kinds of machine before it reaches a service. Each one exists to answer a single question: where should this go, and can I answer it without going further?
flowchart TD U[Browser] -- 1 resolve name --> R[Recursive resolver] R -- 2 returns an IP --> U U -- 3 HTTPS to that IP --> E[Nearest edge PoP: TLS and CDN cache] E -- 4 cache miss --> LB[Load balancer] LB --> GW[API gateway] GW --> S1[Service A] GW --> S2[Service B] S1 --> DB[(Data)]
DNS: turning a name into an address
DNS is a distributed, cached directory that maps a hostname to IP addresses through a chain of lookups. It runs before anything else, so its latency and its answers shape every request that follows.
sequenceDiagram autonumber participant B as App and stub resolver participant R as Recursive resolver participant H as Root and TLD servers participant A as Authoritative server B->>R: api.example.com? (nothing cached locally) R->>H: root: who serves .com? H-->>R: the .com servers R->>H: .com: who serves example.com? H-->>R: the authoritative servers R->>A: A record for api.example.com? A-->>R: 203.0.113.10, TTL 60 R-->>B: 203.0.113.10, cached for 60 s
- The application asks the operating system's stub resolver, which checks its own cache and then forwards to a recursive resolver (the ISP's, or a public one such as 8.8.8.8 or 1.1.1.1).
- The recursive resolver walks the hierarchy: a root server points to the servers for the top-level domain, the TLD server points to the domain's authoritative name servers, and those return the record.
- Every answer carries a TTL. Each hop caches for that long, which is why most lookups never leave the resolver, and why a DNS change takes "up to the TTL" to be seen everywhere.
- DNS normally runs over UDP on port 53, falling back to TCP for large answers. Browsers and phones increasingly use DNS over HTTPS or TLS to the resolver so that the query itself is encrypted.
Routing policies: one name, many answers
An authoritative server does not have to give everyone the same answer. Managed DNS services let you attach a policy to a record:
| Policy | How the answer is chosen | Use it for |
|---|---|---|
| Simple | Same records for everyone | One region, one endpoint |
| Weighted | Proportional split, for example 90/10 | Canary releases, gradual migration |
| Latency-based | The region with the lowest measured latency to the resolver | Multi-region apps |
| Geolocation | By the resolver's country or continent | Data residency, localized content |
| Failover | Primary while its health check passes, otherwise secondary | Active-passive disaster recovery |
| Multivalue | Several healthy IPs, client picks one | Cheap client-side balancing |
- Amazon Route 53 implements exactly this policy list, with health checks that can pull an IP out of the answer within seconds of a failure.
- Cloudflare DNS and Google Cloud DNS serve authoritative zones from anycast networks, so the lookup itself is answered nearby.
- Netflix and other multi-region services use latency-based records so that a user in Singapore lands in an Asian region without any code deciding it.
Geolocation and latency policies see the resolver's address, not the user's. A user in Mumbai using a resolver in Frankfurt gets European answers. EDNS Client Subnet lets resolvers forward part of the client's address to fix this, but not every resolver sends it. Also, a low TTL makes failover fast but multiplies query volume; 60 seconds is a common compromise for endpoints that may move.
- Authoritative
- The server that holds the real records for a zone. Everything else is a cache.
- TTL
- Time to live, in seconds. How long any cache may keep an answer before asking again.
- Zone
- The part of the namespace one authority manages, for example everything under
example.com.
Anycast: one address, many locations
Anycast is a routing trick: the same IP prefix is announced from many data centers, and the internet's routing protocol delivers each packet to whichever announcement is closest in network terms. DNS gives you a single address; anycast makes that address exist everywhere.
flowchart LR
subgraph Users
U1[User in Tokyo]
U2[User in Paris]
U3[User in São Paulo]
end
subgraph Internet["Internet: BGP"]
B((BGP))
end
subgraph PoPs["PoPs"]
P1[Tokyo PoP]
P2[Frankfurt PoP]
P3[São Paulo PoP]
end
U1 --> B --> P1
U2 --> B --> P2
U3 --> B --> P3
Each point of presence tells its upstream networks, using BGP, "I can reach this prefix". Every router on the internet ends up with several paths to the same prefix and picks the one with the shortest AS path, subject to its own policies. "Nearest" therefore means fewest network hops, which usually but not always tracks geography. When a PoP goes down it withdraws its announcement and traffic re-routes to the next one within seconds, with no DNS change and no TTL to wait for.
Anycast fits stateless UDP protocols perfectly, which is why the root DNS servers have used it for twenty years. Long-lived TCP connections are trickier: if routing changes mid-connection, packets can land on a different PoP that has no state for that connection. Large edge networks solve this with stable routing and by keeping connection state very short-lived at the edge, and in practice run all of HTTP over anycast.
- 1.1.1.1 and 8.8.8.8 are single addresses answered from hundreds of locations.
- Cloudflare announces the same prefixes from every one of its data centers; DNS, CDN, WAF and Workers all ride on that.
- AWS Global Accelerator gives you two static anycast IPs that enter the AWS backbone at the nearest edge and are then routed privately to your regional load balancer.
- Google Cloud's global HTTP(S) load balancer is a single anycast IP fronting backends in every region.
- BGP
- Border Gateway Protocol. How networks (autonomous systems) tell each other which IP prefixes they can reach and through whom.
- PoP
- Point of presence. A small data center, usually inside an internet exchange, where an edge network meets local ISPs.
CDN: serving from the edge
A content delivery network is a fleet of caching reverse proxies placed close to users. It answers what it can from the edge and forwards the rest to your origin, so most requests never cross an ocean.
sequenceDiagram
autonumber
participant U as Browser
participant E as Edge PoP
participant S as Origin shield
participant O as Origin
U->>E: GET /app.3f2a.js
alt cache hit
E-->>U: 200 from cache, age 120
else cache miss
E->>S: GET /app.3f2a.js
S->>O: GET /app.3f2a.js (once for all PoPs)
O-->>S: 200, Cache-Control max-age 31536000
S-->>E: 200, stored
E-->>U: 200, stored at the edge
end
What decides a hit
- Cache key. By default the URL, often plus selected headers or query parameters. Two requests with the same key share one cached object. Vary the key on as little as possible; keying on every cookie or header turns the cache off.
- Freshness.
Cache-Control: max-age(every cache) ands-maxage(shared caches only, and it overridesmax-agethere) say how long the object is fresh. After that the edge revalidates withIf-None-Matchand the origin answers304 Not Modifiedif theETagstill matches, which is much cheaper than resending the body. - Stale policies.
stale-while-revalidateserves the old copy instantly and refreshes in the background;stale-if-errorkeeps serving when the origin is down. - Purge. To change something before its TTL expires you invalidate it by URL, tag or everything. Purges take seconds but the safer pattern is versioned URLs: a hash in the file name and a one year TTL, so a deploy changes the URL instead of the content.
# Static asset with a content hash in its name: cache it forever, everywhere
Cache-Control: public, max-age=31536000, immutable
# HTML shell that changes on every deploy: browsers must revalidate, the edge may keep it 60 s
Cache-Control: public, max-age=0, s-maxage=60, stale-while-revalidate=300
ETag: "5f2c9a"
# Personalised API response: never cache in shared caches
Cache-Control: private, no-store
Dynamic content and edge compute
CDNs help even when nothing is cacheable. The TLS handshake completes at a PoP a few milliseconds away, the edge keeps warm, long-lived connections to the origin, and the path across the provider's private backbone is faster than the open internet. On top of that, most CDNs now run code at the edge: authentication checks, A/B assignment, redirects, image resizing, or entire small services, executed in the PoP before anything reaches your servers.
- Amazon CloudFront with an S3 origin is the standard static hosting stack; Origin Shield adds the collapsing layer in Figure 5, and CloudFront Functions or Lambda@Edge run code per request.
- Cloudflare puts the cache, WAF, bot management and Workers in every PoP; Argo routes dynamic traffic over its backbone.
- Akamai and Fastly serve a large share of the web's video and software downloads; Fastly's instant purge is the reference for tag-based invalidation.
- Netflix Open Connect is a private CDN: caching appliances installed inside ISPs so that video never leaves the ISP's network.
Caching a response that contains one user's data under a shared key is the classic CDN leak. Anything personalised must be private or no-store, and the cache key must include whatever makes the response differ. Also, a cache hit ratio is only meaningful per content type: 99 percent on images and 5 percent on API calls average to a number that tells you nothing.
- Origin
- Your actual servers or bucket. The CDN is a cache in front of it.
- Origin shield
- One extra caching layer between all PoPs and the origin, so many edge misses become one origin request.
- Hit ratio
- Share of requests answered from cache. The number that decides how much origin capacity you need.
Load balancers: spreading traffic across machines
A load balancer takes one entry point and spreads its connections or requests over a pool of servers, removing the ones that fail health checks. It is what lets "add another server" translate into more capacity.
flowchart TD C[Clients] --> L4["Layer 4 balancer<br/>sees IP and port, forwards TCP or UDP"] L4 --> L7a["Layer 7 balancer<br/>reads HTTP: host, path, headers"] L4 --> L7b["Layer 7 balancer"] L7a -- /api --> A1[api pool] L7a -- /static --> S1[static pool] L7a -- host admin. --> AD[admin pool] L7b --> A1 A1 -. health check failed .-> X[server removed from pool]
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Sees | Source and destination IP and port | The HTTP request: host, path, headers, cookies |
| Decides per | Connection | Request |
| Can do | Very high throughput, static IPs, any protocol, preserve client IP | Path and host routing, TLS termination, sticky cookies, WebSockets, gRPC, retries, header rewriting |
| Cost | Cheap per packet | Parses every request, so more CPU and latency |
| Examples | AWS NLB, Google Maglev, HAProxy in TCP mode, IPVS | AWS ALB, NGINX, Envoy, HAProxy in HTTP mode, Traefik |
What an L4 balancer forwards is a TCP (or UDP) connection as a whole. How that connection actually works — handshake, windows, congestion — is in the TCP article.
Picking a backend
- Round robin and weighted round robin: simple, fair when requests cost about the same.
- Least connections or least response time: better when requests vary in cost; sends work to the least busy machine.
- Hashing on client IP or a header: the same client lands on the same server. Consistent hashing keeps most of those assignments stable when servers join or leave, which matters for per-server caches. The storage article covers consistent hashing in depth.
- Random with two choices: pick two at random, send to the less loaded. Nearly as good as least connections, with no shared state, so it is popular in distributed balancers.
Health checks, stickiness and draining
Active health checks hit each backend on a schedule (a TCP connect or GET /healthz) and eject it after a few failures; passive checks watch real traffic and eject a backend that returns errors or times out. Sticky sessions pin a client to one backend with a cookie; they are a crutch for servers that keep state in memory and get in the way of scaling, so prefer moving that state out (the next article covers this). Connection draining stops sending new requests to a backend being removed while letting in-flight ones finish, which is what makes zero-downtime deploys possible.
- AWS ships three: NLB (L4, static IPs, millions of requests per second), ALB (L7, path and host rules, WebSockets, gRPC, Lambda targets) and Gateway Load Balancer for third-party firewalls.
- Google Maglev is the L4 balancer under every Google service: software on commodity machines, consistent hashing so a connection survives a balancer failing.
- Kubernetes uses kube-proxy or IPVS for L4 inside the cluster and an Ingress controller (NGINX, Envoy, Traefik) for L7 at the edge.
- HAProxy fronted GitHub and Stack Overflow for years; it is still the reference for what a software balancer can do on one box.
A balancer that terminates TLS or proxies at L7 replaces the client's IP with its own. Backends need X-Forwarded-For (HTTP) or the PROXY protocol (TCP) to know who really connected, and must trust that header only from the balancer, never from the open internet.
Proxies: forward and reverse
A proxy is a machine that makes requests on someone else's behalf. Which side it stands on gives it its name: a forward proxy works for clients, a reverse proxy works for servers. Load balancers, CDNs and API gateways are all specialised reverse proxies.
flowchart TB
subgraph Forward["Forward proxy"]
direction LR
C1[Laptop] --> FP[Corporate proxy]
C2[Laptop] --> FP
FP --> I((Internet))
end
subgraph Reverse["Reverse proxy"]
direction LR
I2((Internet)) --> RP[NGINX or Envoy]
RP --> S1[app 1]
RP --> S2[app 2]
RP --> S3[static files]
end
Forward ~~~ Reverse
A forward proxy sits at the client side's exit. It can enforce policy (which sites are allowed), cache for a whole office, hide the internal network, and give many machines one public address. Cloud NAT gateways and egress proxies are the modern form: private subnets reach the internet through one controlled point with one set of IPs, which is also how you get a fixed IP to whitelist with a partner.
A reverse proxy sits in front of servers. Clients only ever see the proxy, so the real topology is hidden and can change freely. On the way through it can terminate TLS, compress responses, serve static files, cache, rate limit, rewrite headers, and add the security headers every app needs. Almost every production service has one, even if it is only the cloud load balancer.
server {
listen 443 ssl http2;
server_name api.example.com;
location /static/ { root /var/www; expires 1y; }
location / {
proxy_pass http://app_pool; # upstream group of app servers
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_read_timeout 30s;
}
}
Sidecars and the service mesh
Inside a cluster the same idea shrinks to one proxy per service instance, a sidecar. Every call between services passes through two sidecars, which gives you mutual TLS, retries, timeouts, circuit breaking, traffic splitting and per-call metrics without touching application code. The sidecars plus the control plane that configures them form a service mesh. The price is an extra hop per call and a lot of moving parts, so meshes pay off in large polyglot fleets and rarely in a ten-service system.
- NGINX and HAProxy are the reverse proxies most sites start with; Envoy, built at Lyft, is the proxy inside most meshes and many cloud load balancers.
- Istio and Linkerd are the common meshes on Kubernetes; AWS App Mesh and GCP Anthos Service Mesh are the managed versions.
- Cloudflare is, architecturally, a giant reverse proxy that you point your DNS at.
- Squid remains the classic caching forward proxy; AWS NAT Gateway is the cloud-era egress point.
API gateway: one front door for services
An API gateway is a reverse proxy that understands your API. It routes each call to the right service and takes over the cross-cutting work every service would otherwise repeat: authentication, rate limiting, TLS, request validation, and response shaping.
flowchart LR M[Mobile app] --> G W[Web app] --> G P[Partner] --> G G["API gateway<br/>TLS · auth · rate limit · routing · logging"] G -- /users --> U[User service] G -- /orders --> O[Order service] G -- /search --> S[Search service] G -. BFF: one call fans out .-> U G -. and aggregates .-> O
- Routing and versioning. Path, host or header rules map to services;
/v2/can go to a new implementation while/v1/keeps working. - Authentication and authorization. Validate the JWT or API key once at the door and pass a trusted identity header inward. Services still check permissions, but they no longer parse tokens.
- Rate limiting and quotas. Per key, per user or per IP, with
429responses and usage plans for partners. - Protocol translation and shaping. REST outside, gRPC inside; JSON validation against a schema; trimming responses for mobile.
- Backend for frontend. A gateway per client type that fans one call out to several services and aggregates the result, so a phone makes one round trip instead of five.
| Load balancer | API gateway | Service mesh | |
|---|---|---|---|
| Sits | In front of a pool | At the edge of the API | Between every pair of services |
| Knows about | Connections and HTTP basics | Routes, clients, keys, quotas | Service identities and policies |
| Main job | Spread and survive | Protect and simplify the public surface | Secure and observe internal calls |
| Typical products | ALB, NLB, HAProxy | AWS API Gateway, Kong, Apigee | Istio, Linkerd, Consul Connect |
- Amazon API Gateway fronts Lambda and container backends with usage plans, request validation and a WebSocket API mode that keeps connections for you.
- Kong (built on NGINX) and Apigee (Google) are the common self-managed and enterprise gateways; Cloudflare offers the same features at the edge.
- Netflix built Zuul as its gateway and BFF layer so each device type could get responses shaped for it.
A gateway that starts holding business logic becomes a second monolith that every team must deploy through. Keep it to cross-cutting concerns and shaping; the moment it knows what an "order" is, the logic belongs in a service.
Real-time delivery: WebSockets, SSE and long polling
HTTP is request and response; the server cannot speak first. Three techniques give you server push, differing in direction, cost and how much infrastructure has to cooperate.
sequenceDiagram autonumber participant C as Client participant LB as L7 balancer participant S as Server C->>LB: GET /chat, Upgrade: websocket, Sec-WebSocket-Key LB->>S: forwards the upgrade S-->>C: 101 Switching Protocols, Sec-WebSocket-Accept Note over C,S: same TCP connection, now framed messages both ways C->>S: frame: send message S->>C: frame: new message from someone else S->>C: frame: typing indicator C->>S: ping every 30 s, keeps idle timeouts away
| Long polling | Server-Sent Events | WebSockets | |
|---|---|---|---|
| Direction | Server to client, one message per request | Server to client stream | Both ways, full duplex |
| Transport | Plain HTTP requests held open until data or timeout | One HTTP response that never ends, text/event-stream | TCP after an HTTP Upgrade; binary or text frames |
| Reconnect | Client loops | Built into the browser API, with last event id | You write it |
| Works through | Everything | Any HTTP proxy that streams; multiplexes nicely over HTTP/2 | Proxies and balancers that support Upgrade and long idle timeouts |
| Good for | Rare updates, legacy clients | Feeds, notifications, streamed AI responses, progress | Chat, games, collaborative editing, trading |
Scaling connections
A push connection is state: the server holding it is the only one that can write to it. At scale that forces three decisions. The balancer must keep connections open for hours, so idle timeouts go up and health checks must not count long connections as stuck. Messages for a user may originate on any server, so servers subscribe to a pub/sub channel (Redis, Kafka, a broker) and whichever server holds the connection forwards. And a connection registry (user → server) lets you target one user without broadcasting. Managed WebSocket gateways do all three and hand your code plain HTTP callbacks.
- Slack and Discord run chat over WebSockets with gateways that fan out through pub/sub; Discord has written about handling millions of concurrent connections per cluster.
- Firebase Realtime Database and Firestore listeners are WebSockets under the hood, which is why they feel instant on mobile.
- ChatGPT-style streaming uses SSE: the answer is one HTTP response that arrives token by token.
- AWS ALB supports WebSockets natively; API Gateway WebSocket APIs hold the connections and invoke Lambda per message, with the connection id stored in DynamoDB.
Mobile networks and corporate proxies drop idle connections silently. Send application-level pings, expect reconnects, and make every message idempotent or resumable so a reconnect does not duplicate or lose anything. For mobile, a WebSocket held open in the background drains battery; platform push (FCM, APNs) is the right tool when the app is not on screen.
HTTP/1.1, HTTP/2 and HTTP/3 in one minute
The version of HTTP in use changes how many connections a client opens, what a load balancer has to understand, and how much latency the network adds. Every hop in this article negotiates it.
| HTTP/1.1 | HTTP/2 | HTTP/3 | |
|---|---|---|---|
| Transport | TCP, text | TCP, binary frames | QUIC over UDP, TLS 1.3 built in |
| Requests per connection | One at a time (keep-alive reuses the connection); browsers open about six per host | Many streams multiplexed on one connection | Many streams, each independent |
| Head-of-line blocking | At HTTP level: one slow response blocks the next | Fixed at HTTP level, still present at TCP level: one lost packet stalls every stream | Gone: a lost packet stalls only its own stream |
| Extras | Simple, universally supported | Header compression (HPACK), stream priorities | Faster handshake, 0-RTT resumption, connection migration when the network changes |
HTTP/2 still stalls every stream when one TCP packet is lost; that head-of-line blocking is explained in the TCP article, which is also why HTTP/3 moved to QUIC. In practice the edge speaks the newest version to users while talking HTTP/1.1 or HTTP/2 to origins, where latency is low and simplicity wins. Load balancers must understand the version they terminate: an L7 balancer that speaks HTTP/2 to clients can still open one HTTP/1.1 connection per request to backends, and that is normal.
- Cloudflare, CloudFront and Google serve HTTP/3 to browsers by default; most origins still see HTTP/1.1.
- gRPC requires HTTP/2, which is why "does the load balancer support HTTP/2 to the backend" is the first question when adopting it. AWS ALB and Google's load balancers do.
- Mobile apps gain the most from HTTP/3: connection migration keeps a session alive when a phone moves from Wi-Fi to cellular.
Recap
- DNS resolves a name through stub → recursive → root → TLD → authoritative, caching at every hop for the record's TTL. Routing policies (weighted, latency, geolocation, failover) let one name return different answers.
- Anycast announces one prefix from many PoPs over BGP; routing delivers each packet to the nearest announcement and re-routes on failure with no DNS change.
- A CDN is a fleet of caching reverse proxies at the edge. Hits depend on the cache key and
Cache-Control; use hashed file names with long TTLs, purge only for exceptions, and an origin shield to collapse misses. - L4 balancers forward connections by IP and port; L7 balancers read HTTP and can route by path, host and header, terminate TLS and handle WebSockets. Health checks remove bad backends; draining makes deploys safe.
- Forward proxies act for clients (policy, egress, fixed IPs); reverse proxies act for servers (hide topology, TLS, compression, caching). A sidecar proxy per service plus a control plane is a service mesh.
- An API gateway is a reverse proxy that knows your API: routing, auth, rate limits, shaping, BFF aggregation. Keep business logic out of it.
- Long polling, SSE and WebSockets give you server push with rising capability and rising infrastructure demands. Scale push with long idle timeouts, pub/sub fan-out and a connection registry.
- HTTP/2 multiplexes streams on one TCP connection; HTTP/3 moves to QUIC and removes TCP head-of-line blocking. The edge speaks the newest version to users and older ones to origins.
Questions
Try answering each one out loud before opening it. Lead with the one-line answer, then the steps, then one detail from the field.
What happens between typing a URL and the first byte of the response?
The name is resolved to an IP through DNS, the browser opens a TCP or QUIC connection and completes a TLS handshake with the nearest edge, the edge answers from cache or forwards through a load balancer and gateway to a service, and the response comes back the same way.
- DNS: stub resolver, recursive resolver, root, TLD, authoritative; cached at each level by TTL.
- Connection: the IP is usually anycast, so the TCP or QUIC handshake and TLS terminate at a PoP a few milliseconds away.
- Edge: CDN checks its cache; on a miss it forwards over warm connections to the origin's load balancer.
- Origin: L4 then L7 balancing, API gateway for auth and rate limits, then the service and its data stores.
A detail worth adding: with HTTP/3 the connection and TLS handshakes are one round trip, and a returning client can send data in its first packet.
How does DNS resolution work, and where does caching happen?
A recursive resolver walks from the root servers to the TLD servers to the domain's authoritative servers, and every hop caches the answer for the record's TTL.
- The OS stub resolver caches first, then the recursive resolver, which is where most hits happen.
- Root and TLD answers are cached for days, so a cold lookup is rare; the authoritative answer is cached for its own TTL, often 60 to 300 seconds for endpoints that may move.
- Negative answers (no such name) are cached too, which is why a typo in a new record can seem "stuck".
Because caches honour the old TTL, a change is only guaranteed visible after the previous TTL has fully elapsed; lower the TTL ahead of a planned migration.
How would you send users to the nearest region: DNS routing or anycast?
Both work; DNS latency or geolocation routing is simpler and works with ordinary regional IPs, while anycast gives faster failover and a single IP but needs a network you control or a provider that offers it.
- DNS routing decides per resolver, is cached for the TTL, and sees the resolver's location, not the user's.
- Anycast decides per packet at the routing layer, fails over in seconds when a PoP withdraws its route, and gives partners one static IP to allow-list.
- Common combination: anycast at the edge (CDN or Global Accelerator) with regional backends chosen by the provider's own latency data.
AWS Global Accelerator is the packaged version of the second option: two anycast IPs that enter the backbone at the nearest edge and are steered to the healthiest nearby region.
What is anycast, why is it a natural fit for DNS, and what is the catch for TCP?
Anycast is announcing the same IP prefix from many locations so BGP delivers each packet to the nearest one; DNS is a single-packet UDP exchange, so it does not care which location answers.
- "Nearest" means shortest BGP path, which usually but not always matches geography.
- Failover is automatic: a PoP that stops announcing simply disappears from routing tables.
- A TCP connection is state on one machine; if routes shift mid-connection, later packets may reach a PoP that has never seen it and it resets.
Large edge providers run all of HTTP over anycast anyway, because routes are stable for the lifetime of typical connections and the edge keeps connection state short-lived.
What decides whether a CDN serves a request from cache, and how do you keep cached content fresh?
The cache key and the freshness headers decide it: an object is served if a fresh copy exists under the request's key, where fresh is defined by max-age or s-maxage; after that the edge revalidates with the ETag or refetches.
- Keep the key small: URL plus only the headers or query parameters that change the response.
- Use content-hashed file names with a one year
immutableTTL for assets, and short TTLs withstale-while-revalidatefor HTML. - Purge by URL or tag for emergencies, not as the normal way to deploy.
An origin shield collapses simultaneous misses from many PoPs into one origin request, which is what protects the origin during a cache flush or a viral link.
When would you choose a Layer 4 load balancer over a Layer 7 one?
Layer 4 when you need raw throughput, static IPs, non-HTTP protocols or to preserve the client's IP end to end; Layer 7 when routing depends on the request itself or you want the balancer to terminate TLS and handle HTTP features.
- L4 forwards whole connections by IP and port; it never looks inside, so it is fast and protocol-agnostic.
- L7 parses HTTP: host and path routing, header rewriting, sticky cookies, WebSockets, gRPC, retries.
- Large systems use both: an L4 tier absorbs traffic and spreads it across L7 tiers that make routing decisions.
On AWS this is the NLB versus ALB decision; a common pattern is an NLB with static IPs for partners in front of an ALB for path routing.
Compare load balancing algorithms. When does consistent hashing matter?
Round robin for uniform work, least connections for variable work, and hashing when the same client should keep hitting the same server; consistent hashing is the hashing variant that survives servers joining or leaving.
- Weighted variants let bigger machines take more traffic.
- "Power of two choices" gets most of the benefit of least connections with no shared state.
- Plain hashing remaps almost every client when the pool size changes; consistent hashing remaps only about 1/N of them.
Consistent hashing matters when backends hold per-client state or caches, for example a fleet of cache servers or WebSocket servers, and Google's Maglev uses it so a connection survives one balancer dying.
How does a load balancer detect a failed server, and what makes a deploy zero-downtime?
Active health checks probe each backend on a schedule and passive checks watch real traffic; a backend that fails a threshold is ejected, and connection draining lets in-flight requests finish before a backend is removed.
- Health endpoints should check dependencies the request path actually needs, but not so much that a slow database takes every server out at once.
- Ejecting on consecutive failures and re-adding on consecutive successes avoids flapping.
- For a deploy: register the new version, wait until it is healthy, drain the old one, then remove it.
Kubernetes encodes this as readiness probes plus a termination grace period, and the rolling update controller drives the sequence.
What is the difference between a forward proxy and a reverse proxy?
A forward proxy acts on behalf of clients and sits at their exit to the internet; a reverse proxy acts on behalf of servers and sits at their entrance.
- Forward: policy enforcement, shared caching, hiding internal clients, one egress IP. Cloud NAT gateways are the modern form.
- Reverse: hide topology, terminate TLS, compress, cache, rate limit, serve static files. Load balancers, CDNs and API gateways are all reverse proxies.
- Both replace the original address with their own, so both need headers or the PROXY protocol to carry it.
A useful test: who configured it? If the client had to be told about it, it is a forward proxy; if the client has no idea it exists, it is a reverse proxy.
What is a service mesh and how is it different from an API gateway?
A service mesh is a sidecar proxy next to every service plus a control plane, handling security and reliability for internal calls; an API gateway is one proxy at the edge, handling the public API surface.
- Mesh features: mutual TLS between services, retries, timeouts, circuit breaking, traffic splitting, per-call metrics and traces, all without application code.
- Gateway features: routing, client authentication, rate limits, request shaping, BFF aggregation.
- They coexist: the gateway admits a request, the mesh carries it between services.
Meshes cost an extra proxy hop per call and real operational complexity; Istio and Linkerd pay off in large polyglot fleets, not in a handful of services.
What should an API gateway do, and what should it not do?
It should own cross-cutting concerns for the public API: TLS, authentication, rate limiting, routing, validation and response shaping; it should not hold business logic.
- Validating a token once at the door and passing a trusted identity inward removes duplicated auth code from every service.
- A backend-for-frontend gateway can aggregate several service calls into one response for mobile clients.
- Once the gateway knows what an order is, every product change needs a gateway deploy and it becomes a shared bottleneck.
Amazon API Gateway, Kong and Apigee all expose this same feature set, which is a good sign of where the boundary is.
WebSockets, Server-Sent Events or long polling: how do you choose?
Choose by direction and infrastructure: long polling when updates are rare and everything must work, SSE for server-to-client streams over plain HTTP, WebSockets when the client also sends frequently and you can afford stateful connections.
- SSE reconnects automatically, multiplexes over HTTP/2 and passes through any streaming proxy.
- WebSockets are full duplex and binary-capable but need balancers and proxies that support the Upgrade and long idle timeouts.
- Long polling is a loop of ordinary requests held open; simple, but each message costs a request.
Streamed AI responses are SSE; chat, multiplayer and collaborative editing are WebSockets; a notification badge can be either.
How do you scale WebSockets to millions of concurrent connections?
Treat each connection as state pinned to one server, fan messages out through pub/sub so any server can publish, and keep a registry of which server holds which user.
- Balancers need long idle timeouts and must not treat a silent connection as unhealthy; clients send heartbeats.
- Publishers write to a topic (Redis, Kafka, a broker); the server holding the target connection forwards the message.
- Connection servers should be stateless apart from the sockets themselves, so they can be added and drained freely.
Managed gateways such as API Gateway WebSocket APIs hold the sockets and invoke your code over HTTP per message, which turns the problem into ordinary stateless handlers.
What changed from HTTP/1.1 to HTTP/2 to HTTP/3, and why do CDNs care?
HTTP/2 multiplexed many requests over one TCP connection with binary framing and header compression; HTTP/3 moved to QUIC over UDP, which removes TCP's head-of-line blocking, merges the transport and TLS handshakes, and lets a connection survive a network change.
- HTTP/1.1 handles one request at a time per connection, so browsers open several connections per host.
- HTTP/2 still stalls all streams when one TCP packet is lost.
- HTTP/3 gives mobile users the biggest win: fewer round trips on lossy networks and connection migration.
CDNs terminate HTTP/3 at the edge because that is where the lossy last mile is, and talk plain HTTP/1.1 or 2 to origins over reliable links.
Where should TLS be terminated, and why?
As close to the user as possible, usually at the CDN or the L7 load balancer, because the handshake's round trips are cheapest there and it lets everything behind it inspect and route the request.
- Terminating at the edge means certificates and cipher policy live in one place.
- Behind the edge, traffic can be re-encrypted to backends (or carried inside a mesh with mutual TLS) when the network is not trusted.
- Terminating on the backend itself is the L4 pass-through option, used when end-to-end encryption to the application is mandatory.
AWS ALB terminates TLS with ACM certificates and can re-encrypt to targets; NLB can pass TLS through untouched.
Design static asset delivery for a global web app.
Put the built assets in object storage, front it with a CDN, name every file with a content hash and cache it for a year, and serve the HTML shell with a short TTL so each deploy changes the URLs rather than the files.
- Storage: S3 or GCS as the origin, private, with the CDN as the only allowed reader.
- CDN: CloudFront or Cloudflare with an origin shield, compression (Brotli), and HTTP/3 to users.
- Headers:
immutableone-year cache for hashed files;max-age=0, s-maxage=60, stale-while-revalidatefor HTML. - Deploy: upload new hashed files first, then the HTML that references them, so no user ever sees a 404.
Purging is only for a mistake; the hashed names mean a normal release never needs one.