Continuous Integration vs Continuous Delivery

Prefer to watch? ▶ The full episode ✈ Telegram

Almost everybody writes these two words as one word. CI/CD. It gets said like the name of a product you install.

It is not one thing. It is two rules. They are about two different objects, and they point in opposite directions. Most teams follow the first one and stop there, and they do not notice they stopped, because the two words have been glued together for so long.

This article takes them apart.

Merge day

Start with the thing that made you look this up.

Six developers. One codebase. Six branches open at the same time. A branch is your own private copy of the code, where you work without disturbing anyone else. Yours opened on a Monday. You are building one feature, and it touches eighteen files.

While you work, the other five keep working too. Each of them lands one change on the shared trunk — the one main copy of the code that everybody shares — every day. You do not see any of it.

After one day, five changes have landed underneath you. That is fine.

After six weeks, one hundred and fifty changes have landed underneath you. Your branch has not moved. The ground it was standing on has.

Then comes merge day. Nobody put merge day in the plan. Nobody estimated it. But it is a week of work, it happens every single time, and the team has quietly decided that this is simply what software feels like.

What this article covers, and what it does not

Three things are covered:

What continuous integration actually is, and why it is a rule about people rather than a piece of software.
What continuous delivery actually is, and why it is a rule about one object, called the artifact.
The third word, deployment, which is not the same as delivery, and which is where most of the arguments come from.

Three things are not covered. No tool is named anywhere in this article. Not one. How a release is rolled out to users is a different subject, covered elsewhere on this site. And this is not advice about how to write tests.

The workshop

The nearest confusion is this one. People think CI/CD is a single thing. A file in the repository. A list of steps that a machine runs. So they install it, and then they say they have it.

Picture a workshop instead.

On the left there is one bench, with one wooden frame being built on it. Everybody who makes a piece brings it to that bench today, and fits it into the frame today. If two pieces do not fit each other, you find that out today, while each piece is still one cut away from fitting.

On the right there is a loading dock. There is one crate. It is packed, sealed, stamped, and standing at the door. A lorry is waiting outside. Whether the crate leaves is a separate decision, made by a person.

The bench is continuous integration. The crate is continuous delivery. Two different rules, about two different objects.

One instant, two halves

Here is the whole article in one picture.

Draw the life of one change as a line of time. Mark one instant on that line. That instant is the green build: the moment your change has joined everybody else's work, the machine has built it, the tests have run, and everything passed.

Everything to the left of that instant is continuous integration. It looks backwards, at the merge. Its question is: how old is the oldest piece of work that has not joined the trunk yet? A good answer is measured in hours.

Everything to the right of that instant is continuous delivery. It looks forwards, at the artifact. Its question is: how much stands between this build and a real user? A good answer is one button.

Two halves. One instant between them. Pointing in opposite directions. Every section below is this same picture with the marks moved.

What a long branch actually costs

The cost of a long branch is not the cost most people assume. It is worth making concrete, because the arithmetic says something surprising.

Here is a small model of the team.

# drift.py -- what a branch costs is not its size. It is its age.
TEAM          = 6      # developers on this one codebase
REPO_FILES    = 1200   # files that anybody might touch
FILES_I_TOUCH = 18     # what my one branch actually edits

def drift(days_open):
    # every OTHER developer lands one change a day while my branch sits still
    return (TEAM - 1) * days_open

def chance_of_a_conflict(days_open):
    miss = (REPO_FILES - FILES_I_TOUCH) / REPO_FILES   # one edit misses me
    return 1 - miss ** (2 * drift(days_open))          # each change edits 2 files

print(drift(1),  chance_of_a_conflict(1))     # 5 changes    -> 14 percent
print(drift(30), chance_of_a_conflict(30))    # 150 changes  -> 99 percent

Drift is the number of changes that landed on the shared trunk while my branch sat still. Five other developers, one change each per day, so drift is five changes per day.

The chance of a conflict is the chance that at least one of those landed changes touched at least one of my eighteen files. Each landed change edits two files, so I take the chance of one edit missing me, and raise it to that power.

Run it. Every figure below came out of that script, and each one was cross-checked with a 200,000-run simulation that shuffles the edits at random:

How long the branch stayed open	Changes that landed underneath it	Chance of a conflict
1 working day	5	14.0% (simulated: 14.1%)
1 working week	25	53.0%
6 working weeks	150	98.9% (simulated: 98.9%)

Read the first and last rows next to each other. A branch that is one working day old has a 14% chance of hitting a conflict when it merges. Most days, nothing happens. You merge, and you carry on with your afternoon.

A branch that is six weeks old has a 98.9% chance. Not most of the time. Essentially always.

And here is the honest third number. Nothing about the work changed between those two rows. The same eighteen files. The same feature. The same developer. The only thing that changed is that the branch was allowed to live thirty times longer.

Look at the script again and notice what is missing. FILES_I_TOUCH is fixed at eighteen in both runs. The size of the branch never appears in the calculation. Only its age. A branch is a debt, and the interest on it is time.

That is what continuous integration attacks. Not your tests, and not your tooling. The age of the oldest thing that has not been joined yet.

Continuous integration, written out

So here is continuous integration in full. Notice that nothing in it is a product you can buy.

# merge.py -- continuous integration, with nothing bought and nothing installed.

def integrate(my_change):
    trunk  = fetch("trunk")              # 1. what everyone else already landed
    joined = merge(trunk, my_change)     # 2. join my work to it -- TODAY

    if not builds(joined):               # 3. does the whole thing still compile?
        return stop("the build broke")

    if not tests_pass(joined):           # 4. does it still do what it did?
        return stop("the tests broke")

    return push("trunk", joined)         # 5. now everyone builds on my work too

Step by step. Take what everyone else has already landed on the trunk. Join my work to it — not next month, today. Build it: does the whole thing still compile, with my change inside it? If it does not, stop, and say so out loud. Run the tests: does the whole thing still do what it did before? If it does not, stop, and say so out loud.

And if both of those passed, push it back to the trunk. From this moment, everybody else is building on top of my work as well.

That is all of it. Really it is two words. Every change, and today.

A machine that runs your tests but lets branches live for six weeks is not doing this. It is just running your tests.

The part that gets skipped: CI is a rule about people

Let us say this plainly, because it is the part that gets skipped.

Continuous integration is a rule about how people work. The machine is only there to enforce it.

Which means a team can have every green tick in the world and still not be doing it. If the tests run on every proposed change, but those proposals stay open for three weeks, then the joining still happens once every three weeks. The machine is measuring something. It is not fixing anything.

A green tick on a three-week-old branch proves that the branch was fine three weeks ago, against a trunk that no longer exists.

And it is cheap to install, but expensive to keep. Installing it takes an afternoon. Keeping it means the test suite has to stay fast, and it has to stay honest.

The moment people stop believing a red result, you have made things worse than having no tests at all.

The arithmetic of a suite nobody believes

That last sentence sounds like an opinion. It is arithmetic.

A flaky test is one that sometimes fails even though the code is perfectly fine. Say you have four hundred tests. Say each one, on its own, fails wrongly about two times in every thousand runs.

Two in a thousand sounds like nothing at all. But a build is only green when all four hundred pass. Multiply it out:

Wrong failures per test, per 1,000 runs	Chance a perfectly good build shows red	In other words
0.5	18.1%	1 good build in every 5.5
1	33.0%	1 good build in every 3
2	55.1%	1 good build in every 1.8

At two in a thousand, more than half of your perfectly good builds show red. So more than half the time, red means nothing.

And then the number that actually costs you: zero. That is how many people will look carefully at a red build, once they have learned that red usually means nothing. They press the button that runs it again.

Now a real failure and a fake failure look exactly the same. Your suite has stopped being a signal. It is still running, it is still costing machine time, and it is no longer telling anybody anything.

This is why "we installed it" is not the same as "we have it". Continuous integration is a habit that a machine happens to enforce, and habits decay.

Crossing the line: the artifact

Now cross to the right hand side of that instant, and follow one green build all the way out.

Here is the green build. The change is on the trunk, it compiled, the tests passed. Continuous integration has done its whole job, and it stops right here.

Continuous delivery starts here. The first thing it does is turn that build into one object. One package, with one identity, built exactly once. That object is called the artifact, and from this moment it is the only thing that moves.

The same artifact is then pushed into real environments. Not a rebuild. The same bytes, with different configuration — addresses, secrets, sizes — handed to it from outside.

And at the end there is production, with one step in front of it. In continuous delivery, that step is a button, and a person presses it.

Build once, promote the same bytes

Here is what that means in code. This is the rule that carries the whole idea.

# build.py -- the artifact is built ONCE. Only the configuration changes.

def build_artifact(commit):
    # one build, one identity. This object is the only thing that ever ships.
    return package(commit, digest=sha256(commit))

def promote(artifact, environment):
    config = load_config(environment)    # address, secret, size -- from OUTSIDE
    return deploy(artifact, config)      # the same bytes. Nothing is rebuilt.

art = build_artifact("a91f4c")
for env in ("staging", "pre-production", "production"):
    promote(art, env)                    # 1 build, 3 deploys, identical bytes

# Rebuild once per environment instead and you get 3 builds -- which is
# 2 chances for what reaches production to differ from what you tested.

Build the artifact from one commit. Package it, and give it an identity — a digest, which is a short fingerprint of the contents, so you can point at exactly this thing and no other.

Promoting is a separate step, and this is the important line. Promoting takes the artifact and an environment. It loads that environment's configuration from outside the artifact. Then it deploys the artifact, unchanged.

So one build goes to staging, then to pre-production, then to production. Three deployments, one set of bytes. The thing your users get is the exact thing your tests ran against.

Now do it the other way. Build again for each environment. Three builds, which means two chances for what reaches production to be quietly different from what you tested. Nobody chooses that on purpose. It happens because rebuilding is the easy thing to write.

What continuous delivery demands

Continuous delivery is not free, and the price is not money. It is a list of things that have to already be true.

Required:

One artifact for every green build, and nothing else ever ships.
Configuration comes from outside that artifact, always.
Database changes that work with both the old and the new version of the code, because for a few minutes both of them are running.
A rollback you have actually used, on a real day. Not a paragraph in a document.

And here is what has to go:

Servers that somebody edits by hand.
A separate build for each environment.
A long-lived release branch.
A freeze week before every release.
A page of manual steps written down in a document.

Without that list, continuous delivery is only a faster way to ship the same bug.

One boundary, so you know what this is not

There is a whole subject sitting next to this one, and it is worth naming so you can see the edge.

How a release is rolled out is that other subject, and it is covered elsewhere on this site: two full production stacks with the traffic flipped between them, a small share of users getting the new version first, and requests already in flight being allowed to finish before the old version stops.

Shipping code switched off and turning it on later is a third subject, also covered elsewhere, and it is not the same as either half here.

All of them are real, and all of them matter. This article is about something before all of them: how often work is joined together, whether every green build becomes one artifact, and how far that artifact travels on its own.

The pipeline, and the artifact. Not the rollout. This article stops at the artifact.

The third D

And now the third word, which is where almost every argument about CI/CD actually comes from.

Continuous delivery means every green build is ready to go to production. It has been built once, it has been through the environments, and it is sitting at the door. A person decides when it goes. The team can ship ten times a day if they want to, and they can also decide not to ship anything for a week. Ready is the promise. Released is a choice.

Continuous deployment removes the person. Green means it goes. Nobody presses anything.

Both of them are written CD, which is exactly why people argue past each other. One person is defending a discipline, the other is defending an automation, and they both think the letters mean the same thing.

They are also not the same difficulty at all. Delivery needs discipline. Deployment needs everything delivery needs, plus a way to notice a bad release and undo it without a human — because there is no longer a human in the loop to notice.

The difference is often one line

Inside a real pipeline, the difference between those two is often exactly one setting.

# promote.py -- continuous DELIVERY and continuous DEPLOYMENT, in one file.

REQUIRE_A_HUMAN = True      # True: delivery.   False: deployment.

def on_green_build(artifact):
    promote(artifact, "staging")
    if not smoke_tests("staging"):
        return stop("staging is unhappy -- nothing goes anywhere")

    if REQUIRE_A_HUMAN:
        return wait_for_the_button(artifact)    # DELIVERY: ready, not released
    return promote(artifact, "production")      # DEPLOYMENT: it ships itself

At the top there is one setting: require a human, true or false.

On a green build, promote the artifact to staging, and run the quick checks there. If staging is unhappy, stop, and nothing goes anywhere.

And now the branch. If require-a-human is true, you wait for the button. The artifact is finished, tested, and standing still. That is continuous delivery: ready, but not released.

If require-a-human is false, the same artifact goes to production right there, on the same green build, with nobody watching. That is continuous deployment.

Continuous delivery ends at a button that nobody is obliged to press. Continuous deployment deletes the button.

Note what did not change between the two paths: the artifact, the promotion, the checks. Everything above that setting is the same work. That is why you cannot skip delivery on the way to deployment.

Why the size of a release matters, on a bad day

There is one more reason to care, and it only shows up when something goes wrong.

Suppose your six developers finish one change each per day, and you release once a quarter. That release contains 378 changes. Then something breaks in production. You now have 378 suspects. Finding the one, by repeatedly cutting the list in half, takes about 9 rounds.

Now release once a day instead. That release contains 6 changes. Six suspects. About 3 rounds.

Release rhythm	Changes in one release	Rounds of halving to find the culprit
Once a quarter	378	~9
Once a day	6	~3
Difference	63×	3×

The same team. The same code. The same bugs. The only thing that changed is how much work was allowed to pile up between one release and the next.

This is not just arithmetic on a page, either. The multi-year Accelerate research found the same direction in real organisations: teams that release in small batches, often, also recover faster and fail less. Small batches are not a nicety on top of the pipeline. They are most of what the pipeline buys you.

The same two halves, when part of what you ship is a model

If your system includes a trained model, both halves still apply, and one of them gets harder.

The left half does not change at all. Prompts, retrieval settings, evaluation scripts and serving code are all just code on a trunk, and a branch holding them is exactly as expensive with age as any other branch.

The right half is where people quietly break the rule. The artifact stops being one object. The code is packaged once, but the model weights are fetched at start-up from wherever the latest ones happen to live, and the prompt text is edited in a console by hand. Now three things ship on three different schedules, and none of them has one identity. When output quality drops on a Tuesday, you cannot say what changed, because "what changed" was never a single thing you could point at.

The fix is the same sentence as before, applied honestly: one artifact per green build. Pin the weights by digest, put the prompt in the package, and let configuration — endpoints, sizes, keys — come from outside. Then a bad Tuesday has one identity attached to it, and rolling back means going back to one object rather than reconstructing three.

Three questions about your own team

You can answer these this afternoon.

First. How old is the oldest piece of work that has not been joined to the trunk yet? If you answer in weeks, you do not have continuous integration, no matter what runs on your proposed changes.

Second. Does every green build produce exactly one artifact, which is never rebuilt on its way out? If not, you do not have continuous delivery yet — because there is nothing being delivered.

Third. What is the last step before production? If it is a button, that is delivery. If there is no step at all, that is deployment. If it is a person following instructions written in a document, then it is neither of them.

The verdict

	Continuous integration	Continuous delivery
It is a rule about	the merge	the artifact
It looks	backwards, at joining work	forwards, at shipping it
Its question	How old is the oldest unjoined work?	How much stands between this build and a user?
A good answer	hours	one button
What it costs	cheap to install, expensive to keep honest	discipline: config outside, two-way migrations, a used rollback
How it fails quietly	green ticks on three-week-old branches	a rebuild per environment

Continuous integration is a rule about the merge. Everybody's work joins the same trunk today, and every join is built and tested, so a break is found while the change is still small enough to understand.

Continuous delivery is a rule about the artifact. Every green build becomes one package, built once, that travels the environments unchanged, until it is standing one press away from production.

And the D that everybody argues about is a third thing. Delivery means always ready. Deployment means the person is gone as well.

You can have the first without the second, and almost everybody does.

References and further reading

On branch age and merge day

Paul Hammant and contributors, Trunk Based Development (trunkbaseddevelopment.com) — the short-lived-branch rule and why branch lifetime, not branch size, is the thing to manage: trunkbaseddevelopment.com

On continuous integration as a rule about people

Kent Beck, Extreme Programming Explained: Embrace Change (Addison-Wesley, 1999; 2nd ed. 2004) — the origin of "integrate continuously" as a team practice rather than a piece of infrastructure.
Martin Fowler, Continuous Integration (martinfowler.com, 2006) — the canonical definition, including "everyone commits to mainline every day": martinfowler.com/articles/continuousIntegration.html

On a suite people stop believing

Martin Fowler, Eradicating Non-Determinism in Tests (martinfowler.com, 2011) — why a suite that fails at random destroys the trust that makes it worth running at all: martinfowler.com/articles/nonDeterminism.html

On the artifact, and build-once-promote

Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley, 2010) — the deployment pipeline, building binaries exactly once, and keeping configuration outside the artifact.

On the third D: delivery versus deployment

Martin Fowler, ContinuousDelivery and Continuous Delivery vs Continuous Deployment (martinfowler.com) — the ready-versus-released distinction stated directly: martinfowler.com/bliki/ContinuousDelivery.html

On batch size and release rhythm

Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (IT Revolution, 2018) — the measured link between small batches, deployment frequency, lead time and recovery.

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Continuous Integration vs Continuous Delivery on YouTube