Monowai

writing - continuous integration

CI is a practice, not a product

Enterprise CI/CD in regulated environments. Part two of three - Continuous Integration.

Mike Holdsworth · Singapore

Part one argued that the approval gate is not the problem: a regulated organisation can do real Continuous Delivery, and the thing to be skeptical about is manual labour either side of the approval. That article split delivery into two numbers. This one is about the first of them - lead time to releasable, the number that belongs entirely to your team.

Continuous Integration is a series of practices that promote the continuous integration of work into a common branch. That sounds circular until you sit with it: the practice is the merging. The workflow file automates the checking. It does not do the integrating. Merge queues and auto-merge on green automate the landing too, and they are worth having. They do not decide how small the change is or how long it sat on a branch first - and that is the part that makes integration continuous.

This is where "I wrote a GitHub workflow, therefore we're doing CI/CD" falls apart. The file is real work and worth doing. It is also the part a competent team finishes in a week. Everything that makes CI hard sits upstream of it.

Branch age is a status you should be tracking

A branch is isolated by definition. From the moment it's cut it drifts from the common source of truth, and every hour it lives the drift compounds. Branching seems to be a practice poorly understood by teams.

To borrow the RAG status: any branch lasting longer than a day should be considered Amber, and longer than two days is Red. A long-lived branch prevents progress being made across the team - it hides work, and it defers the integration cost to the worst possible moment, when everybody else has moved on too.

Why do we get them? The most common reason I see is epics masquerading as stories. The feature is far too big and it went unsaid. The branch is a symptom; the story sizing is the disease.

There's a second reason specific to regulated shops: the long-lived release branch that exists "for compliance". It doesn't serve compliance. What your control framework wants is traceability from commit to production and an evidenced approval - and a branch that has diverged for six weeks makes both of those harder to produce, not easier. The version and the evidence come out of the pipeline. They don't come out of the branching strategy.

Honestly? Anything that benefits the team and improves the product should be merged at the earliest opportunity - library upgrades, refactoring in the name of clarity, improved test coverage. Waiting doesn't make any of them safer. Under regulation it makes them considerably less safe, because the change that eventually reaches the gate is now enormous, and "large change" is the single best predictor of a bad night.

"But the feature isn't finished"

The first objection, every time, and a fair one. Merging daily cannot mean exposing half a feature to users. So the practices that make small merges possible aren't optional extras - they are what makes CI achievable at all. Without them, "merge every day" is advice no team can actually take.

Feature flags. Integrate the code, keep it dark, switch it on independently of the deployment. Release stops being a deployment event and becomes a configuration decision.

Two costs, and the second one catches regulated teams out. Flags are inventory - every one is another branch in the code and another combination in your test matrix, and removing them has to be part of the work rather than a tidy-up ticket that never gets picked up. And a flag flip in production is a change. Either it sits inside your change control process or you have just found a hole in it. Decide that deliberately, before somebody decides it for you during an audit.

Branch by abstraction. For changes too structural to flag - swapping a persistence layer, replacing a framework. Introduce a seam, move callers onto it one at a time, build the new implementation behind it, then delete the old path. Every step is independently mergeable. None of them is a six-week branch.

Expand and contract. The same move for schemas and contracts. Add the new column, field or endpoint; backfill; migrate the readers; then remove the old one. Several small deployable changes instead of one breaking change. This is also what preserves your ability to roll back, which part three picks up.

None of these are advanced techniques. All of them are the difference between a team that integrates continuously and a team that talks about it.

What a CI workflow actually guarantees

The steps vary with your technical situation. The guarantee shouldn't: every push produces a versioned, scanned, evidenced artefact that no human touched.

State it as a guarantee rather than a checklist, because the checklist is what people copy and the guarantee is what people skip. The steps that usually get you there:

  • It compiles and it's honest - lint, build, test, code coverage
  • It has been looked at - static analysis across your source, composition analysis across your dependencies, a scan across the image you built. Three different questions with three different answers. Teams collapse them into "code scanning", then wonder which one found nothing
  • It exists as a thing - packaged artefact, packaged image, published version. Immutable, addressable, identified by digest rather than by a tag somebody can move
  • It can prove it - test results, scan results, provenance and approvals attached to the artefact, not to a wiki page
  • It asks to be deployed - submitted for deployment as a request, with its evidence travelling with it

Evidence is the artefact's passport

That last point deserves its own heading, because it's what changes a regulated team's life.

Evidence is not a report you assemble at the end for somebody else's benefit. It's produced once, at build time, and carried everywhere the artefact goes. Test results, scan results, coverage, the commit it came from, the pipeline identity that built it.

One requirement follows from this and is easy to miss: evidence has to be machine-deliverable, not just machine-produced. If it can only be read by a human opening a dashboard, somebody will end up exporting it by hand - and you've reintroduced the toil at the last step. Emit it in a form the pipeline can carry onward without assistance.

Do that and your audit position stops being an end-of-cycle scramble and becomes a query. Somebody asks "what was in release 4.7 and who checked it", and the answer is a lookup rather than a fortnight of archaeology. It also means the approver at the gate has something to read that took no one two days to prepare - which is the difference between a four-minute approval and a four-day one.

The Golden Pipeline trap

Teams should have effective workflows that automate the work they'd otherwise be doing locally and manually. What organisations should avoid is the trap of the "Golden Pipeline" - one central pipeline, centrally owned, mandatory for everyone.

The failure mode is always the same. Team A upgrades their Node or their Gradle tooling. The central team, for reasons that have nothing to do with Team A's application, doesn't support that version and doesn't have capacity to add it this quarter. Team A is now blocked on somebody else's backlog. This is the easiest way for things to go out of step.

In regulated organisations the trap has a stronger form, and it's the one worth watching for: the central pipeline is mandated as a control. Now you can't argue with it, because arguing looks like arguing with compliance.

But read the control objective. It says changes must be built, scanned, approved and traceable. It does not say every team must use build system version 4.2. A paved road that produces attested, evidenced outputs satisfies that objective completely. A mandated single version satisfies nothing except the central team's roadmap, and it creates a systemic risk while doing so: every team's ability to patch a critical vulnerability now depends on one queue.

The distinction worth holding onto:

  • A paved road is optional, versioned and forkable. A team can pin an older version, or step off and accept the cost of evidencing compliance another way. The central team ships a library, not a gate.
  • A golden pipeline is mandatory and single-versioned. The central team ships a bottleneck.

Same artefact, opposite operating model. This kind of lockstep is not an accelerator to what is essentially a pretty simple task.

The number this owns

Lead time to releasable: commit to an approved-and-evidenced, promotable artefact, sitting there waiting for somebody to decide.

Under an hour is achievable in almost any technical situation. No change window applies to it, no CAB governs it, and no approver sits in its path. If it's measured in days, that's yours to fix - and no amount of pointing at the governance process will make it somebody else's.

Deployment frequency into non-production belongs here too, for the same reason: your test environments are not subject to anyone's release calendar. A team that can't deploy to its own test environment on demand hasn't been blocked by regulation. It's been blocked by something it chose.

Part three: what happens after the build - promoting one artefact through environments, what a good gate looks like, freezes, and rollback as a pre-approved path.

References & further reading

  • Continuous Delivery - Humble & Farley (2010)
  • Trunk Based Development - Hammant (2020)
  • Feature Toggles - Hodgson / Fowler (2017)
  • Branch by Abstraction - Fowler (2014)
  • Refactoring Databases: Evolutionary Database Design - Ambler & Sadalage (2006)
  • Accelerate - Forsgren, Humble & Kim (2018)

← All writing