Fixing Uneven Load Distribution with Passthrough Routes and Persistent Connections

Applications that use SSL passthrough routes or ClusterIP Services with persistent HTTP connections often experience uneven request distribution across pods. This applies to all Red Hat managed OpenShift services, including ARO, ROSA, and OSD on GCP. This guide explains why that happens, reproduces the problem in three scenarios, and demonstrates a fix for each.

The Problem

OpenShift handles passthrough routes at Layer 4 (TCP). HAProxy balances TCP connections, not individual HTTP requests. When a client opens a persistent (keep-alive) connection through a passthrough route, all requests on that connection go to the same backend pod.

The same behavior applies to internal traffic through a ClusterIP Service. OVN-Kubernetes uses a 5-tuple hash (source IP, source port, destination IP, destination port, protocol) to select a backend. A single persistent connection always produces the same hash, so every request on that connection reaches the same pod.

With a small number of long-lived clients (connection pools, sidecar proxies, or batch jobs), only a few pods handle the bulk of traffic while others sit idle.

Why This Matters

Setting haproxy.router.openshift.io/balance: roundrobin and haproxy.router.openshift.io/disable_cookies: "true" on a passthrough route does not help. These annotations control how new TCP connections are assigned, not how requests within a connection are routed. Under sustained load with connection pooling, a single client sends all of its requests to one pod while others receive none. In production with many concurrent clients, some pods can end up handling significantly more traffic than others.

Prerequisites

• An ARO, ROSA, or OSD on GCP cluster (or any OpenShift cluster)
• oc CLI logged in with permissions to create namespaces, deployments, services, and routes

Setup

Create a namespace and deploy a single echo server that listens on two ports: TLS on 8443 for the passthrough route scenario and plain HTTP on 8080 for the ClusterIP scenarios. Each pod returns its hostname in the response so you can see which pod handled each request.

Wait for the rollout to complete:

Scenario 1: External Traffic via Passthrough Route

Create the Service and Passthrough Route

Reproduce the Problem

Get the route hostname and run 5 parallel clients, each sending 200 requests over a single persistent TLS connection:

Check the results:

Expected output: each client pins all 200 requests to a single pod. With 5 clients, only 3 of 10 pods receive traffic.

Fix: Switch to an Edge Route

An edge route terminates TLS at HAProxy and forwards plain HTTP to the backend. This lets HAProxy inspect HTTP traffic and balance at Layer 7, distributing individual requests across pods rather than pinning entire connections.

Run the same load test against the edge route:

Expected output: with an edge route, HAProxy balances at Layer 7. All 10 pods receive traffic from each client, compared to the passthrough test where each client pinned 100% to a single pod.

The distribution is approximately even (7%-13% per pod) rather than mathematically perfect, because HAProxy’s L7 roundrobin accounts for backend response times. The key difference is that all 10 pods are active.

Scenario 2: Internal Traffic via ClusterIP Service

Create a ClusterIP Service

Reproduce the Problem

Reset the echo server and run 5 parallel clients that each send 2,000 requests over a single persistent HTTP/1.1 connection through the ClusterIP Service. The client uses a raw socket to guarantee that the same TCP connection (and therefore the same OVN-K 5-tuple hash) is used for every request:

Expected output: each client sends all 2,000 requests to a single pod. With 5 clients and 10 pods, half the pods receive zero traffic.

Fix: Headless Service with Client-Side Load Balancing

A headless Service (clusterIP: None) does not proxy traffic. Instead, DNS returns the IP addresses of all backing pods. The client resolves these IPs and distributes requests across them directly.

Reset the echo server and run the headless load test:

The following load test opens a new connection to a different pod IP for each request. This is the simplest way to demonstrate DNS-based distribution. In production, client-side load-balancing libraries (Spring Cloud LoadBalancer, gRPC name resolver, Netflix Ribbon) maintain a pool of persistent connections spread across all pod IPs, which achieves even distribution without the overhead of a new connection per request.

Expected output: every client distributes requests evenly, exactly 10.0% per pod across all 10 replicas.

Each of the 5 clients shows the same perfectly even distribution across all 10 pods.

Scenario 3: Internal Traffic via an Internal IngressController

When internal services call a backend like a decision server, modifying the client application to use DNS-based round-robin (Scenario 2) is not always practical. An alternative is to route internal traffic through an internal IngressController with an edge route, so HAProxy handles L7 balancing without any client code changes.

Create an Internal IngressController

Wait for the IngressController to become available:

Create an Internal Edge Route

Create an edge route with the router: internal label so it is served by the internal IngressController. The edge termination is critical: it lets HAProxy balance at Layer 7 per request instead of per connection.

Test Internal Traffic Through the Route

Internal callers send traffic to the route hostname instead of the ClusterIP Service name. The internal IngressController resolves within the cluster, so no external DNS or egress is needed.

Get the internal router’s cluster IP and the route hostname:

Reset the echo server and run the load test. The client resolves the route hostname via the internal router’s IP to ensure traffic stays in-cluster:

Expected output: all 10 pods receive traffic from each client, compared to the ClusterIP test where each client pinned 100% to a single pod.

As with the edge route in Scenario 1, the distribution is approximately even rather than perfect. A pod that shows a very low percentage (for example, 0.5%) typically just started and had not yet passed HAProxy’s backend health checks when the test began. The key result is that all 10 pods are active and receiving traffic. Internal callers get L7 per-request balancing without any changes to the client application.

Summary

Cleanup