--- title: "Feature Flags vs Load Balancer: One Splits Versions, the Other Splits People" description: "Both tools split your traffic, so 'send 10% to the new thing' sounds like one task. It is two, at two different layers. A load balancer splits VERSIONS by weight at the infrastructure layer — 90/10 at Envoy, Istio or NGINX — and it splits connections, not people, so the same user can bounce between v1 and v2 on every request. A feature flag splits BEHAVIOR inside one running service, per user, by hashing a stable id into a bucket so the same person always lands in the same group. Here is the code for both, what each one actually costs you, the rule of thumb, and how the same split shows up in an LLM serving stack." keywords: "feature flags vs load balancer, feature flag, load balancer, canary deployment, blue-green deployment, gradual rollout, A/B testing, traffic splitting, sticky sessions, Envoy, Istio, NGINX weights, hash bucketing, LLM model rollout, system design" created_at: "2026-08-19T12:30:00" post_type: "anonym_post" content_type: "technical_article" --- ![Banner](./banner.webp)

Prefer to watch? ▶ The 2-minute version ✈ Telegram

Someone says *"just send 10% of the traffic to the new version."* It sounds like one task with one obvious tool. It is actually two completely different tools, sitting at two different layers of your stack — and reaching for the wrong one is why a rollout that should take an afternoon turns into a week of fighting your own infrastructure. The one-line mental model: - A **load balancer** splits **versions**. It sits in front of your deployments and sends *connections* to one of them by weight. - A **feature flag** splits **behavior**. It sits *inside* one running service and decides, per user, which code path executes. Both of them can produce the sentence "10% are on the new thing." They mean very different things by it. --- ## The load balancer: weights in front of deployments A load balancer splits at the **infrastructure layer**, between whole deployments of a service. You run v1 and v2 side by side, and you tell the proxy how to divide traffic between them. In Envoy, Istio, or NGINX that is a weight: ```yaml # Istio VirtualService — a 90/10 canary spec: hosts: ["checkout"] http: - route: - destination: host: checkout subset: v1 weight: 90 - destination: host: checkout subset: v2 weight: 10 ``` That single block is your **canary**. Turn it to 100/0 and 0/100 and it is **blue-green**. Walk it 90 → 75 → 50 → 0 over an afternoon and it is a **gradual rollout**. Point the two subsets at two builds of your frontend and it is an **edge A/B test**. All of it is the same primitive: a number next to a destination. The appeal is that it is completely **outside your application**. The service code has no idea any of this is happening. There is no `if` anywhere, no library to import, and you can roll back a bad deploy by editing one integer — the old version is still running, untouched, waiting. ### The catch: it splits connections, not people Here is the part that surprises people the first time it bites them. **A load balancer does not know who your user is.** It sees a connection, applies a weight, and picks a pod. That means: - The **same user can land on v1 now and v2 on the next request**. If v2 changed the shape of an API response or the layout of a page, that user just watched the app flicker between two versions. The fix is **sticky sessions** — pinning by cookie or by hashed source IP — and that is an extra thing to configure, plus it skews your split and complicates draining. - You **cannot target a segment**. "Only beta users," "only Germany," "only accounts on the Pro plan" — none of these are expressible as a weight, because the proxy has no user model. Sticky sessions make a user *consistent*; they still do not make the user *selectable*. - **Changing the split is a config change.** A reconfigure, a redeploy, a pipeline run. Not a toggle you flip in ten seconds when the error rate moves. --- ## The feature flag: a bucket per user, inside the service A feature flag splits at the **application layer** — inside a single running service, on a per-request, per-user basis. And the trick at the heart of it, the one that turns up in interviews constantly, is **hash bucketing**: ```python import hashlib def bucket(user_id: str, buckets: int = 5) -> int: """Stable bucket for a user: same id -> same bucket, forever.""" digest = hashlib.sha256(user_id.encode()).hexdigest() return int(digest, 16) % buckets def new_checkout_enabled(user_id: str) -> bool: return bucket(user_id, 5) == 0 # bucket 0 = 20% of users ``` Read that carefully, because every property you want falls out of it: - **It is a percentage.** Five buckets, one of them enabled, is 20% of your users. Want 10%? Change the `5` to a `10`. Want 50%? Two buckets out of four, or `bucket < 5` out of ten. - **It is stable.** The hash is a pure function of the user id, so the same person hashes to the same bucket on every request, on every pod, after every restart, forever. No sticky sessions, no cookies, no shared state — you get consistency for free because you derived the group from the user instead of from chance. - **It is deterministic across services.** Hash the same id with the same function in three different services and all three agree on which group that user is in, without talking to each other. Do not use a random number here. `random.random() < 0.2` also produces "20% of traffic," but it re-rolls on every single request — the user gets the new checkout, then the old one, then the new one again. That is the load balancer's problem, reproduced in your own code. Once behavior is keyed to the user rather than to the connection, the things a weight could never express become trivial: ```python def new_checkout_enabled(user, flags) -> bool: if user.id in flags.beta_list: return True # explicit allowlist if user.plan != "pro": return False # segment targeting if user.country not in ("DE","CH"): return False # geo targeting return bucket(user.id, flags.buckets) < flags.on # everyone else: % ``` And because the rule is data, not deployed code, you flip it in a dashboard and it takes effect on the next request. **No redeploy.** That is the real reason flags win incidents: your rollback is a toggle, not a pipeline. ### The catch: every flag is an `if` you now own Flags are not free either, and their cost lands somewhere less visible than the load balancer's. - **Both code paths ship in the same binary.** The new code is already deployed and already running — it is just hidden behind a condition. A flag is not isolation. A memory leak or a bad import in the "off" branch is in production the moment you deploy it. - **Stale flags rot.** Every flag is a live branch in your code, and a flag nobody dared delete two quarters ago is now a dead path that every future change has to keep compiling. Two flags interacting is four states to reason about; ten flags is a combinatorial mess nobody has tested. Flags need an expiry date and someone whose job is removing them. - **A flag cannot move load.** This is the boundary. If a pod is saturated, a flag does not help — the request still arrives at the same overloaded process. Shedding load, draining an instance, surviving an availability zone: that is the load balancer's job and only the load balancer's job. --- ## The same split, in an LLM serving stack This distinction gets sharper, not blurrier, when the thing you are rolling out is a model — and it is where most teams building on LLMs first hit it. You have a new model version, or a rewritten system prompt, or a retrieval pipeline with a different chunking strategy. You want 10% of traffic on it. **Which layer?** **Route versions at the gateway.** A new model *weight* — a fine-tune, a quantization, an upgraded provider version — is a deployment. It has its own container, its own GPU memory, its own batching behavior and its own throughput profile. Splitting that by weight at the gateway is exactly the load balancer's job, and the reason is capacity: you are not just testing quality, you are testing whether the thing survives contact with real concurrency. A weighted split lets you watch **TTFT, tokens per second, and GPU utilization** on the new replica set under genuine load, and cut it to zero in one config change if the p99 falls apart. A per-user flag cannot do this, because a flag does not control which GPU serves the request. **Gate behavior with a flag.** The prompt, the retrieval strategy, the temperature, whether this user gets the tool-calling agent or the plain completion — all of that is behavior inside one service, and it must be **keyed to the user**, for a reason that goes beyond consistency: your **evaluation depends on it**. If you are comparing satisfaction, task completion or thumbs-up rate between two prompts, a user who silently drifts between variants mid-session poisons both arms of the experiment. The bucket hash is what makes the comparison valid — every event that user emits is attributable to exactly one variant, for the whole test. The clean setup uses both at once, and they do not overlap: **the gateway decides which model process serves you; the flag decides what you ask it.** --- ## The comparison

	Load balancer	Feature flag
Splits	Versions (deployments)	Behavior (code paths)
Layer	Infrastructure, in front	Application, inside
Unit of the split	Connection / request	User (stable hash bucket)
Same user, same result?	Only with sticky sessions	Yes, by construction
Target a segment?	No — a weight has no user model	Yes — plan, country, allowlist
Change the split	Reconfigure / redeploy	Dashboard toggle, instant
New code runs in prod?	Only on the new version's pods	Always — both paths are shipped
Can it shed load?	Yes, that is its purpose	No — same pods either way
Long-term cost	Config sprawl in the proxy	Dead branches nobody deletes
Good for	Canary, blue-green, rollback, capacity	A/B tests, betas, kill switches, entitlements

--- ## The verdict The rule of thumb fits in one line: > **Splitting versions of a service → load balancer. Splitting behavior for specific users → feature flag.** If the two things you are choosing between are **two builds**, the split belongs outside your code, at the proxy, as a weight. If they are **two behaviors of the same build**, the split belongs inside your code, keyed to the user, as a bucket. For **A/B tests specifically, reach for the flag** — almost always. Not because the load balancer cannot divide traffic, but because an A/B test is a measurement, and a measurement needs the variant tied to a *person* and to your analytics events. A random pod assignment gives you a split; it does not give you a result you can trust. And the two compose, which is what a good rollout actually looks like: **canary the deployment at the load balancer** so a broken build only reaches 10% of connections and can be pulled with one integer, and **gate the features inside it with flags** so you choose who experiences what, and can kill any of it in ten seconds without shipping anything. One last thing worth internalizing, because it is the question behind both tools: *what is the unit you are splitting?* A load balancer's unit is a connection, and connections are anonymous and interchangeable. A flag's unit is a person, and people notice when the software changes underneath them. Pick the tool whose unit matches the thing you actually care about keeping consistent. --- **Want the short version?** The [2-minute breakdown is here](https://youtube.com/shorts/6AM-h5Rasa8) — both layers, the bucket hash, and the rule of thumb, animated.