Circuit Breaker — Explained in Detail

Prefer to watch? ▶ Full walkthrough (9 min) ▶ 2-minute summary ✈ Telegram

Your checkout service is down. Nobody deployed anything. The payment provider didn't crash—it just got slow. Your engineers reach for the obvious fix: retry harder. Thirty seconds later, all three retry attempts time out. Meanwhile, your worker pool is exhausted waiting on that slow dependency, and the whole checkout system collapses. Retrying a slow service isn't a cure; it's an accelerant.

Mental model: A circuit breaker is a fuse that stops hammering a slow dependency before your own system drowns in its own retries.

The Naive Approach: Why Retrying Kills You

Here's what broken looks like:

def charge(customer_id, amount):
    for attempt in range(3):
        try:
            return payment_provider.charge(
                customer_id,
                amount,
                timeout=30  # seconds
            )
        except TimeoutError:
            if attempt == 2:
                raise
            continue

You see a timeout. You retry. The payment provider is still slow, so you wait another 30 seconds. You retry again. Total: 90 seconds of blocked execution per failed charge.

Now consider the traffic math. If you're processing roughly 1 request per second during peak checkout, and each failed charge costs 90 seconds of worker time, you've just burned 90 worker-seconds per second of incoming traffic. If your worker pool has 10 workers, you're out of capacity within seconds. New requests pile up. The queue grows. Your system becomes unresponsive not because the payment provider crashed, but because you're retry-hammering it while your own workers suffocate.

The problem: you're treating latency as a recoverable error.

The Three States

A circuit breaker sits between your code and the dependency. It tracks failures and decides whether to even attempt the call.

Closed (normal operation): Requests flow through. Failures are counted. When failures exceed a threshold within a time window, the breaker trips.

Open (dependency is dead to us): New requests fail immediately without calling the dependency. No timeout. No retry. Fail fast. The breaker remembers when it opened so it can probe later.

Half-Open (cautiously testing recovery): After a reset timeout has passed, the breaker allows a single test request through. If it succeeds, the breaker closes. If it fails, the breaker reopens and waits another reset period.

This is the shape of a real system: you stop pretending the dependency is fine, fail fast, and let the dependency breathe while you downgrade gracefully elsewhere.

Building It: State, Thresholds, and the Trip Logic

Here's a minimal working breaker:

import time

class CircuitBreaker:
    def __init__(
        self,
        fail_max=5,
        reset_timeout=60
    ):
        self.fail_max = fail_max
        self.reset_timeout = reset_timeout
        self.state = 'closed'
        self.failures = 0
        self.opened_at = None
        self.last_failure_time = None

    def call(self, func, *args, **kwargs):
        now = time.time()

        # If half-open, let one request through
        if self.state == 'half-open':
            try:
                result = func(*args, **kwargs)
                self._on_success()
                return result
            except Exception as e:
                self._on_failure(now)
                raise

        # If open, check if reset window has passed
        if self.state == 'open':
            if now - self.opened_at > self.reset_timeout:
                self.state = 'half-open'
                return self.call(func, *args, **kwargs)
            else:
                raise Exception(
                    f"Circuit breaker open; "
                    f"retry in {self.opened_at + self.reset_timeout - now:.1f}s"
                )

        # Closed: attempt the call
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure(now)
            raise

    def _on_failure(self, now):
        self.failures += 1
        self.last_failure_time = now
        if self.failures >= self.fail_max:
            self.state = 'open'
            self.opened_at = now

    def _on_success(self):
        self.failures = 0
        self.state = 'closed'

The core logic: count failures; when you hit the threshold, switch to open and stop calling the dependency; after the reset timeout, cautiously try once more.

Real Usage: Charge with a Fallback

The breaker only stops the bleeding. You still need to answer the customer.

breaker = CircuitBreaker(fail_max=5, reset_timeout=60)

def checkout_handler(customer_id, amount):
    try:
        charge_id = breaker.call(
            payment_provider.charge,
            customer_id,
            amount,
            timeout=30
        )
        return {"status": "charged", "id": charge_id}
    except Exception as e:
        # Breaker is open, or charge failed.
        # Queue it; answer pending.
        queue.enqueue(
            "process_charge_later",
            customer_id,
            amount
        )
        return {"status": "pending", "message": "We'll charge you shortly."}

When the breaker opens, you don't burn worker cycles retrying. You queue the charge for later (background job, async retry with backoff), return a pending response, and free the worker to handle other requests. The user sees a message like "We're processing your order" instead of a timeout.

Meanwhile, the slow payment provider recovers. After 60 seconds, the half-open state lets a probe through. One request succeeds, the breaker closes, and the queued charges drain.

Where It Breaks at Scale

Circuit breakers assume a single breaker instance or shared state. With 20 replicas of your checkout service, you have 20 independent breaker instances.

Scenario: Payment provider is slow. Replica A hits fail_max, opens its breaker. Replicas B–T don't know. They keep hammering the provider. The provider gets 19/20ths of the traffic still piling on. It drowns anyway.

Solution: Shared state. Write breaker state to Redis or a central store so all replicas see the same decision.

class SharedCircuitBreaker(CircuitBreaker):
    def __init__(self, redis_client, key, fail_max=5, reset_timeout=60):
        super().__init__(fail_max, reset_timeout)
        self.redis = redis_client
        self.key = key

    def call(self, func, *args, **kwargs):
        # Fetch state from Redis
        state_data = self.redis.get(self.key)
        if state_data:
            self.state = state_data['state']
            self.failures = state_data['failures']
            self.opened_at = state_data['opened_at']

        try:
            result = super().call(func, *args, **kwargs)
        finally:
            # Persist state back to Redis
            self.redis.set(
                self.key,
                {
                    'state': self.state,
                    'failures': self.failures,
                    'opened_at': self.opened_at
                }
            )
        return result

Now all 20 replicas see the same breaker state. The first to detect a failure causes all to respect it.

Circuit Breaker in LLM Inference

If you're serving LLM inference, think of a circuit breaker around the model-serving endpoint, not around individual tokens. When your inference service gets slow (high TTFT—time to first token—or throughput drops), a circuit breaker prevents request queuing from pushing tail latency past SLO. Instead of retrying inference requests into a slow queue, trip the breaker and shed load by rejecting new inference requests or routing them to a fallback model. This is cleaner than trying to guess whether the slowness is recoverable.

Approach	Behavior Under Slow Dependency	Worker Pool Impact	When to Use
Naive Retry	Blocks and retries 3× (90s total if 30s timeout)	Exhausts all workers; pile-on effect	Never for external dependencies; only for transient network hiccups on fast operations
Timeout Only	Waits 30s, fails, returns error	Blocks one worker per request; manageable if dependency recovers quickly	When failure rate is low and you have enough worker capacity
Circuit Breaker	Fails instantly after threshold; no timeout burn	Frees workers immediately; prevents pile-on	When dependency is external, unreliable, or prone to cascading slowness

The Whole Thing in One Breath

A dependency gets slow. Retrying makes your own system collapse by exhausting worker pools. A circuit breaker stops retrying, fails fast, and moves the charge to a queue or fallback. It opens (stops calling the dependency), waits for recovery, then cautiously probes with a half-open state. At scale, you need shared state so all replicas respect the same breaker decision. The payoff: your workers stay free, your system stays responsive, and the dependency gets the breathing room to recover.

Verdict

Reach for a simple timeout-only approach when your dependency is internal, fast, and rarely fails. Reach for a circuit breaker when the dependency is external, slow-prone, or shared across many replicas—it stops retries from becoming worse than the original outage.

Watch the 90-second reel for the visual walkthrough of the three states and the failure trace.