60 Years of Data Storage: Database → Warehouse → Lake → Lakehouse

Ask an engineer whether S3 is a data lake and most will say yes. It isn't — and the reason is the entire sixty-year history of how we store data.

We have rebuilt that stack four separate times: database, warehouse, lake, lakehouse. Nobody did it for fashion. Each generation exists because the one before it failed at exactly one thing. Once you know what broke, the buzzwords stop being buzzwords and collapse into a single argument.

• Mental model: every generation traded one constraint for another. Relational bought flexibility with scan cost. The warehouse bought speed with rigidity. The lake bought flexibility back with chaos. The lakehouse bought order back with a metadata layer.

To keep it concrete, follow one online bookstore the whole way down.

---

Before 1970: The Question Was Welded Into the Structure

Our bookstore's orders live as bytes on a disk. Somebody has to decide how those bytes are arranged so a program can find anything.

In the navigational era — IMS shipped in 1966, built for the Apollo program — the arrangement was a hierarchy of records connected by hard pointers. You didn't ask a question. You traversed: start at the root, follow this pointer, then that one, read the record.

That works beautifully for the question the structure was designed around, and it is useless for any other. The access path was baked into the file. A new question meant a new program.

1970: Codd Separates WHAT From HOW

Edgar Codd's paper — A Relational Model of Data for Large Shared Data Banks — proposed something that sounds obvious now and was radical then: stop hard-wiring the path.

Store flat tables. Relate them by value, not by pointer: orders.book_id matches books.id because the values match, not because someone stored an address. Then let the caller describe what they want and let the system work out how to get it.

SELECT b.title, SUM(o.total)
FROM   orders o
JOIN   books  b ON b.id = o.book_id
GROUP  BY b.title;

System R at IBM (1974) turned this into SEQUEL, then SQL. Ingres came out of Berkeley around the same time. Oracle v2 shipped in 1979.

The win is that the bookstore can now ask a question nobody planned for. The query optimizer, not the application, decides the access path.

The Crack: One Table Doing Two Jobs

Success created the next problem.

The same orders table now serves two workloads with opposite shapes:

• Checkout (OLTP): touch one row, write it, return in milliseconds. Thousands of tiny concurrent transactions.
• The monthly report (OLAP): scan every row that ever existed, aggregate, return one number.

They are on the same disk, competing for the same buffers and the same locks. Run the report and customers wait behind it. Worse, a row-store has to read whole rows off disk even when the report wants two columns out of forty.

You cannot tune one database for both. That is not a configuration problem — it is a physics problem.

1988–1996: The Data Warehouse

So analytics moved out. The term "business data warehouse" comes from Devlin and Murphy at IBM in 1988; Inmon's Building the Data Warehouse (1992) and Kimball's The Data Warehouse Toolkit (1996) turned it into a discipline.

The shape is:

1. Extract from the operational databases, nightly.
2. Transform — clean, conform, deduplicate, resolve keys.
3. Load into a separate store, modelled as a star schema: a central fact table (one row per order line) surrounded by dimension tables (date, book, store, buyer).

This is schema on write. You decide the structure before the data lands. Column stores like Vertica (2005) pushed it further by storing each column separately, so a query touching two columns reads two columns.

It works — and it makes a bet: you already know the questions.

Why the Bet Stopped Paying

Then the bookstore launches a website, and the data stops looking like rows.

Clickstream events arrive as JSON whose shape changes whenever a frontend developer ships. Server logs have no schema at all. Cover images and support-call audio aren't rows in any useful sense. The transform step rejects all of it, because it only accepts the shape it was designed for.

And there is a second, quieter problem: in a classic warehouse, storage and compute are welded together in the same appliance. Keeping three more years of history means buying CPU you don't need, because the box only scales as one unit.

2003–2010: Hadoop and the Data Lake

Google published the Google File System paper in 2003 and MapReduce in 2004: spread files across a thousand cheap machines, assume some are always dead, and ship the computation to the data instead of pulling the data to the computation. Yahoo's Hadoop (2006) made it available to everyone.

In 2010 James Dixon, then CTO of Pentaho, coined the contrast that named the era: a data mart is bottled water — cleansed, packaged, structured for consumption. A data lake is the lake itself: the raw contents, in their natural state.

So flip the bet. Dump the raw bytes into storage, cheaply, in a columnar file format like Parquet, and decide the schema when you read. Keep everything, plan nothing.

Why Most Lakes Became Swamps

The flip removed the constraint that was doing real work.

A folder is not a table. Concretely, what a lake was missing:

• No catalog. Nothing records which files are the orders table. You end up with orders/, orders_v2_FINAL/, and orders_backup_july/, and tribal knowledge about which is real.
• No transactions. A job writing new files gets read halfway through, so two dashboards compute two different revenue numbers from "the same" data — and there is no way to tell which is right.
• No schema enforcement. Schema-on-read means every reader re-implements the interpretation, and they drift.
• Slow. Listing millions of objects to plan a query is its own bottleneck.

What actually survived the Hadoop era wasn't Hadoop. It was S3 (launched March 2006) and Spark (started at Berkeley's AMPLab in 2009). S3 made storage cheap, durable and independent of any cluster; Spark made the compute layer pleasant enough to use. Between them, storage and compute finally came apart.

So — Is S3 a Data Lake?

No. S3 is bytes. A data lake is a stack, and the bucket is only the bottom of it:

Layer	What it does	Example
Object storage	durable bytes, cheap, independent of compute	S3, GCS, R2
File format	columnar layout, compression, column pruning	Parquet, ORC
Table format	which files are the table, right now, atomically	Iceberg, Delta, Hudi
Catalog	names, permissions, discovery	Glue, Unity, Polaris
Engine	actually runs the query	Spark, Trino, DuckDB

Calling a bucket a data lake skips the three layers that make it a table rather than a pile of files. Nearly every "our data lake became a swamp" story is a stack that stopped at row two.

2012–2014: The Warehouse Steals the Good Idea

While lakes were struggling, the warehouse absorbed the one genuinely great idea of the lake era. Redshift (2012), BigQuery, and above all Snowflake (out of stealth 2014) separated storage from compute: one copy of the data, many independent compute clusters, each sized and billed on its own. Snowflake's 2020 IPO was the largest software IPO at the time.

Now you had two credible stacks — and most companies ran both.

The Two-System Tax

Which is its own failure mode:

• Two copies of the data, and a nightly job to sync them.
• Two permission models, so "who can see salary data" has two answers.
• Two engines and two bills.
• And when the numbers disagree, a week of somebody's life spent finding out why.

2016–2019: Table Formats Are the Missing Layer

Iceberg started at Netflix in 2016 (Apache in 2018), Hudi at Uber, and Delta Lake was open-sourced by Databricks in April 2019. They solve exactly the layer that was missing for a decade.

A table format is a metadata tree that sits on top of ordinary Parquet files:

orders/
  data/
    part-0001.parquet
    part-0002.parquet
  metadata/
    v2.metadata.json      ← the current table
    snap-8471.avro        ← manifest list
    manifest-0003.avro    ← which data files, with stats

The metadata file lists precisely which data files constitute the table right now. That single indirection buys you:

• Atomic commits. A writer stages new files, then swaps one pointer. Readers see the old table or the new one, never half of one.
• Schema evolution. Add, drop or rename a column by ID, without rewriting the data.
• Time travel. The previous pointer still resolves, so VERSION AS OF is just reading an older metadata file.
• Fast planning. Per-file statistics let the engine skip files without listing the bucket.

That is the whole trick: not a new storage system, a new description of one.

2020: The Lakehouse

Put a catalog and a query engine on top of that and the second system becomes unnecessary. Databricks coined "lakehouse" in 2020 (the CIDR paper landed in 2021): one set of open files in your own bucket, with the sales dashboard and the recommendation training job reading the same table.

The ownership bargain is genuinely different from 1990. The file format and the table format are public specifications, so the tables outlive whichever vendor you're paying this year — which is exactly the property the appliance era did not have.

The format war ended in convergence: in June 2024, days after Snowflake announced Polaris, Databricks acquired Tabular (the company behind Iceberg). The industry has effectively settled on open table formats as neutral infrastructure — and moved the lock-in question up one layer, to the catalog.

What To Actually Choose

The genuinely useful part, and the part nobody sells you:

Situation	Use
Under ~100 GB	One Postgres. Or DuckDB on a laptop.
Terabytes, questions you can't name yet	Object storage + Parquet + Iceberg
Heavy BI on questions you already know	A warehouse still wins
Both, at scale	Lakehouse

Hardware moved while the architecture discussion did not. A single machine with 128 GB of RAM and DuckDB will beat a small Spark cluster on 100 GB of Parquet, and it will beat it on cost by an order of magnitude.

The most expensive mistake in this field is still building the lake when one Postgres would have carried you three more years. Every generation above was invented by someone who had genuinely outgrown the previous one. Adopt the failure before you adopt the fix, and you inherit all of the complexity and none of the reason.