Monitoring vs Observability: Why Every Dashboard Is Green During the Outage

Three in the morning. Your checkout is broken. Every dashboard glows green.

That is not a bug in your dashboards. It is what a dashboard is: a question somebody asked in advance, frozen into a picture. It answers that one question forever. When tonight's failure was never on the list, ten panels have nothing to say.

• Mental model: Monitoring is a smoke alarm. Observability is the floor plan. One tells you the house is on fire; the other lets you figure out why.

---

The Shape of the Problem

Monitoring and observability are not the same thing, and they are not substitutes for each other. You need both.

Monitoring covers the things you already knew to worry about: error rate, latency, queue depth, CPU, memory. You build these dashboards before an incident happens. They are cheap to run, they work, and you should keep them.

Observability is for the questions nobody wrote down yet. Only customers on the new plan. Only the two machines you deployed at 2pm. Only uploads over 5 MB. Only requests from the EU. Real incidents start there, because the ones you charted, you already fixed.

The distinction is not about tidier logs, vendor shopping, or alert fatigue. Those are symptoms. The distinction is this: known unknowns versus unknown unknowns.

---

Known Unknowns vs Unknown Unknowns

Before an incident, you know what you don't know. You know you should watch error rate. You know you should watch latency. You know you should watch queue depth. These are known unknowns. You ask the question, you build the dashboard, you set the alert. Monitoring.

During an incident, you encounter unknown unknowns. The failure only affects requests with plan: enterprise and a content_type: video. You did not know you needed to worry about that combination. Your dashboards have no answer. Observability.

Monitoring tells you that something is wrong. Observability tells you what is wrong and why.

---

The Mechanism: Stop Aggregating at Write Time

The shift from monitoring to observability is a shift in when you lose detail.

With traditional monitoring (metrics), you aggregate at write time. A request comes in, you bump a counter: http_requests_total{method="POST", status="200"}. The detail is already gone. You cannot ask about this request later because you threw away everything except method and status.

With observability (events), you emit one wide event per request with everything on it:

{
  "timestamp": "2024-01-15T03:47:22Z",
  "user_id": "user_9847",
  "plan_tier": "enterprise",
  "region": "us-west-2",
  "content_type": "video",
  "content_size_bytes": 524288,
  "request_duration_ms": 2847,
  "status": 200,
  "cache_hit": false,
  "retries": 2,
  "shard_id": "shard_03"
}

Nothing was aggregated. The detail is still there. During the outage, you get to invent the question: "Show me requests where plan_tier is enterprise AND content_type is video AND status is not 200." You did not have to ask that question in advance.

---

The Cardinality Trap

This is where teams break their own systems.

The temptation is to bolt these fields onto your metrics as labels. One metric, every dimension:

http_requests_total{method, status, user_id, plan_tier, region, content_type, shard_id}

Math: 5 methods × 10 statuses × 50,000 users × 4 plan tiers × 3 regions × 8 content types × 10 shards = 480,000,000 unique series. Your metrics database will not survive. Your write path collapses. You are now less observable, not more.

The rule: Metrics add up at write time. Events add up at read time.

With metrics, the cardinality explodes the moment you write the first request. With events, you store them raw. The cardinality only matters when you aggregate during the query, and you only aggregate what you asked for.

Aspect	Monitoring (Metrics)	Observability (Events)
Aggregation timing	Write time	Read time
Detail retention	Chosen in advance	Every field preserved
Cardinality risk	High (explodes on write)	Managed (only at query)
Cost model	Cheap writes, expensive storage	Expensive writes, cheaper queries
Answers unknown questions	No	Yes

---

For LLM Inference

If you are running an LLM service, apply the same split. Monitoring gives you token throughput, time-to-first-token, time-per-output-token, model inference latency. These are your smoke alarms—set them up today.

Observability events capture the request shape: prompt length, context window usage, batch size, model variant, quantization, temperature, max_tokens, actual output length. When latency spikes only for 8K-token prompts at 2am, you need those fields in your events, not summarized away in a metric. You cannot ask that question in advance because you did not know that combination mattered until the incident told you.

---

The Verdict

Reach for monitoring (dashboards, metrics, known alerts) when you have already fought the battle and won—you know what to watch. Reach for observability (events, high-cardinality logging, post-hoc queries) when you need to ask questions during the outage that nobody wrote down yet.

You need both. Monitoring keeps the lights on. Observability keeps you sane at 3am.

Watch the full walkthrough with code at https://youtu.be/NG2hRJN9UHI, or the 3-minute summary at https://youtu.be/TEIWFRIxlLo.