L4/L7 load balancing, proxies, WebSockets and draining

Engineering focus: “How do packet-forwarding L4 load balancers, TCP proxies and HTTP reverse proxies differ? Choose routing strategies for short HTTP and long-lived sessions, explain AWS/GCP paths, then show how Kubernetes ingress, health checks and draining prevent deployment outages.”

A load balancer distributes incoming work across eligible backends, but the most useful distinction isn't simply L4 versus L7. Ask who owns the TCP connection, which protocol is interpreted, where TLS terminates, and which control plane updates healthy endpoint membership.

Visual architecture / protocol flow

The Mermaid diagram illustrates a representative path; the accompanying explanations cover implementation-specific caveats.

1. Three handling models: packet forwarding, TCP proxy, HTTP proxy

A packet-forwarding L4 device selects targets using connection/flow information and the backend remains the TCP peer. A transport proxy terminates the frontend TCP connection and starts a separate backend TCP connection while it may relay encrypted TLS bytes. An HTTP reverse proxy understands methods, host/path/headers and HTTP status, normally terminating frontend HTTP/TLS. TLS termination is distinct from TCP termination: a TCP proxy can relay an encrypted TLS stream without decrypting it. L4 products may have basic TLS/SNI-aware features without offering arbitrary HTTP path routing.

2. Forward proxy, reverse proxy, API gateway and mesh

A forward proxy acts on behalf of clients, often enforcing outbound policy and handling HTTPS through CONNECT. A reverse proxy acts on behalf of backend servers, presenting a common public endpoint and choosing upstream services. An API gateway adds API-level authentication, quotas, validation and transformations. A service mesh distributes service-to-service policy (mTLS, retries, authorization, observability) across workloads. These roles overlap in products, but responsibilities differ. When using X-Forwarded-For or Proxy Protocol for client IP, define which proxies are trusted; client-supplied forwarding headers are forgeable.

3. Request routing versus endpoint selection

An HTTP route rule maps Host/path/header matching to a logical backend pool. Within the pool, a balancing policy selects a healthy endpoint. The proxy can reuse upstream connections or create new ones. HTTP/2 permits many streams on one frontend connection; the proxy may translate to HTTP/1.1 backend connections or use another protocol. One frontend HTTP/2 connection does not imply one backend connection or that all requests go to the same pod.

4. Choosing balancing algorithms

Algorithm	Uses	Blind spot
Round-robin	Sequence of assignments	Doesn't account for per-request duration
Weighted round-robin	Relative configured capacity	Weights aren't exact business-request fractions
Least connections	Fewer active backend sessions	Idle vs expensive session cost may differ
Least outstanding requests	Fewer in-progress HTTP operations	Doesn't directly measure bytes/CPU
Power of two choices	Picks less loaded of two sampled endpoints	Quality depends on chosen load signal
Source IP hash	Source-address affinity	NAT causes many users to share a key
Consistent/rendezvous hash	Stable partitioning/cache locality	Doesn't migrate established sessions
Locality-aware routing	Region/zone affinity	Can concentrate failure or capacity imbalance

For long-lived WebSockets, active connection count is often informative; for request/response APIs, in-flight request count and latency may be better. Existing connections do not automatically redistribute when replicas scale up. Affinity is not state replication.

5. AWS ALB and NLB

ALB is HTTP-aware: HTTPS listener, certificate/TLS policy, host/path rules, target groups and health checks. It can speak HTTP/1.1 or HTTP/2 to clients and uses separately configured backend protocols. EKS controller integration can register instances/NodePorts or pod IP targets. ALB supports WebSockets. Documented ALB target quota defaults can change and may be adjustable; never present a historical quota as immutable. Current AWS published ALB default is 1,000 targets per ALB (adjustable; verify region/account).

NLB is a network/transport-oriented service. TCP forwarding chooses a target for a flow; a TCP listener can pass TLS bytes to a downstream terminator, while a TLS listener terminates TLS at NLB. The target selection and connection lifecycle differ from ALB. NLB's default TCP flow idle timeout is documented as 350 seconds for relevant TCP listeners; check listener-specific settings. Keep backend deregistration and availability-zone placement in mind.

6. GCP load balancer families and GKE NEGs

Google Cloud distinguishes Application Load Balancers (L7 HTTP proxies), proxy Network Load Balancers (terminate and establish distinct TCP connections), and passthrough Network Load Balancers (packet forwarding; direct-server-return characteristics). A GCP Application LB is assembled from forwarding rule → target HTTPS proxy → URL map → backend service/health check → instance group or zonal NEG.

In container-native GKE L7 ingress, NEGs can register pod IP:port directly. Kubernetes readiness and Google Cloud backend-health/NEG readiness are distinct states. A pod may be Ready but not yet registered or healthy at the LB. For Service type=LoadBalancer, GKE normally uses an appropriate L4 design, not automatic HTTP L7 NEG routing.

7. HAProxy HTTP and TCP modes

HAProxy mode http parses HTTP and supports host/path ACLs, request logging, backend HTTP checks and HTTP routing. mode tcp relays transport bytes; absent TLS termination, it cannot inspect encrypted URL paths. A frontend accepts traffic, an ACL/rule chooses a backend, and backend server entries represent actual endpoints. For Kubernetes, dynamically changing pod IPs require controller-driven endpoint discovery or supported dynamic configuration rather than manually hardcoding thousands of server lines.

global
    log stdout format raw local0
    maxconn 20000

defaults
    mode http
    log global
    option httplog
    timeout connect 5s
    timeout client 30s
    timeout server 30s
    timeout tunnel 1h

frontend public_http
    bind *:8080
    acl is_checkout path_beg /checkout
    use_backend checkout if is_checkout
    default_backend catalog

backend checkout
    balance leastconn
    option httpchk GET /health
    server a 10.20.1.10:8080 check
    server b 10.20.1.11:8080 check

backend catalog
    balance roundrobin
    server a 10.20.2.10:8080 check

The addresses are documentation examples. HAProxy backend queues, time-to-connect, connection errors, session termination codes, socket/file descriptor usage and TLS overhead are essential observability dimensions.

8. NGINX HTTP proxying and WebSocket upgrade

WebSocket over HTTP/1.1 upgrades a request using Upgrade: websocket and Connection: Upgrade, with server response 101 Switching Protocols. These hop-by-hop headers require suitable proxy behavior. NGINX typically needs proxy_set_header Upgrade $http_upgrade and an appropriate Connection value; proxy_read_timeout matters for idle tunnels. WebSocket over HTTP/2 and HTTP/3 follows different mechanisms (extended CONNECT), so 101 is not universal.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}
server {
    listen 8080;
    location /ws/ {
        proxy_pass http://websocket_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 3600s;
    }
}

9. Envoy routing, connection pools and resilience

Envoy configures listeners → filter chains/HTTP connection manager → routes → clusters → endpoint sets. xDS can distribute dynamic listeners/routes/clusters/endpoints. HTTP/1.1 and HTTP/2 upstream connection pools differ: multiplexing can dramatically reduce backend connection count. Circuit-breaker limits prevent unbounded outstanding work; outlier detection can eject failing endpoints. Retry budgets and idempotency matter: a proxy retry after a response is lost can replay an operation that actually committed. Never add retries to payment-like writes without application safeguards.

10. WebSockets and connection scale

A WebSocket maintains a long-lived bidirectional session. Backend selection usually happens at connection/upgrade time, not on every frame. Scaling up adds slots for new sessions; existing TCP sessions remain attached to their original destinations. Scaling down is riskier: the chosen pod may have many active sessions. Monitor active sessions/pod, connection creation, reconnect rate, message throughput, bytes, file descriptors, memory and event-loop responsiveness—not only CPU. A large client reconnection wave can simultaneously overload TLS handshakes, auth, caches and downstream services. Reconnect with bounded exponential backoff and jitter.

11. Active/passive health versus Kubernetes readiness

Active LB health probes can be TCP, HTTP or supported application checks. Passive health may use real connection or response failures. Kubernetes readiness means an endpoint is eligible under its pod probes/gates; LB health is the LB's distinct view. An endpoint can be healthy for /health but fail all checkout transactions due to database saturation. A poor health check may remove every backend when a shared dependency has a temporary outage, worsening availability.

12. Graceful draining and rolling updates

A safe rollout must coordinate new pod ready → new endpoint routable → old endpoint marked draining → stop new admission → finish/close work → process exit. Controller/NEG/target deregistration propagates asynchronously. preStop consumes time inside terminationGracePeriodSeconds; it doesn't automatically drain requests. Cloud backend drain delay, ingress timeouts, endpoint readiness, application shutdown and client retries all need a coherent plan.

spec:
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  template:
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: api
        image: example/api:1.0
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
          periodSeconds: 5
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "wget -qO- http://127.0.0.1:8080/drain; sleep 20"]

This is illustrative and needs a real drain endpoint and suitable container tooling. WebSocket sessions may outlive any practical drain window; clients need reconnection/resume mechanisms. A PDB doesn't protect against all failures or guarantee session survival.

13. Troubleshoot each hop and error owner

kubectl get ingress,gateway,svc -A
kubectl get endpointslices -A
kubectl describe pod POD -n NAMESPACE
ss -lntp && ss -ti
sudo tcpdump -i any -nn 'tcp port 443'
haproxy -c -f /etc/haproxy/haproxy.cfg
nginx -t
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
gcloud compute backend-services get-health BACKEND --global

Distinguish 502 proxy/backend-invalid response, 503 unavailable/routing/no eligible backend, 504 proxy upstream timeout, and application-generated 5xx. Exact semantics depend on the product. Correlate frontend and backend connection tuples and time buckets. A proxy can bottleneck on max connections, upstream queue, descriptors or backend latency despite low CPU.

14. Key takeaways

Distinguish L4 packet forwarding, L4 TCP proxying and L7 HTTP reverse proxying; the differences are connection ownership, protocol visibility and where TLS terminates. HTTP routing selects a logical service by host/path and a separate algorithm selects a healthy backend. Round-robin works for similar requests, least outstanding for variable HTTP work, and least connections can suit long-lived sessions, but no algorithm guarantees even CPU. On AWS, ALB provides L7 listener rules and target groups, while NLB forwards TCP flows or terminates TLS with a TLS listener. On Google Cloud, Application LBs use target HTTPS proxies, URL maps and backend services, while proxy and passthrough NLBs have different TCP semantics. GKE container-native ingress can route directly to NEG pod IP:port endpoints. For WebSockets, established sessions don't rebalance on scale-out, and scale-down requires draining and reconnect logic. Coordinate pod readiness, endpoint propagation and LB target health during rollouts. To debug an outage, identify the exact 5xx producer and measure frontend connection, backend connection, queue, endpoint health, application latency and dependency behavior rather than immediately adding replicas.