Sharding and partitioning

How do you choose a shard key?

By the access pattern, not by the data. It has to spread writes evenly, and it has to be present on the queries you actually run — pick a key the common query does not filter on and every read fans out to every shard, which is worse than one big database. Monotonic keys like a timestamp or an auto-increment id are the classic trap: they put every new write on the last shard.

The follow-up: what happens to queries that need to cross shards, and what happens to transactions. Both get much harder, and that is the real cost of sharding.

What is consistent hashing, and what problem does it solve?

With plain `hash(key) % N`, changing the number of nodes changes the answer for nearly every key — adding one server rehomes almost the entire dataset. Consistent hashing puts nodes and keys on a ring and assigns each key to the next node clockwise, so adding or removing a node moves only the slice between neighbours. Virtual nodes are then added per physical machine so the slices come out even.

When would you not shard?

Almost always, for as long as possible. Sharding costs you cross-shard joins, distributed transactions, rebalancing, and an operational surface that has to be right at three in the morning. Read replicas, better indexes, caching and a bigger machine buy years, and modern hardware goes further than most people assume. Shard when a single primary genuinely cannot absorb the write volume — that is the one problem the alternatives do not solve.