Monitoring, observability and tracing

What is the difference between monitoring and observability?

Monitoring answers questions you thought of in advance: you defined the dashboards and the alerts, and they tell you when a known thing breaks. Observability is whether you can answer a question you did not anticipate, from the data you already emit, without shipping new code. Every dashboard being green during an outage is the signature of a system that is monitored and not observable.

The practical test: can you ask "which customers on this plan saw errors in this region in the last ten minutes" without a deploy? If not, your telemetry is too aggregated.

Logs, metrics and traces — when do you reach for each?

Metrics are cheap numbers over time — they tell you *that* something is wrong and are what you alert on. Traces follow one request across services and tell you *where* the time or the error went. Logs are the detail of a single event and tell you *why*. The workflow is metric to trace to log, from cheap and aggregate to expensive and specific — and the thing that makes it work is a request id present in all three.

Why alert on p99 rather than the average?

Because the average hides everyone who is having a bad time. A service with a 40 ms mean and a 4-second p99 is failing one request in a hundred badly enough for the user to leave, and the mean will never show it. The tail is also what compounds: a request that fans out to ten services sees the tail of every one of them, so a p99 in a dependency becomes a common experience upstream.