CI That Stops Rebuilding the World: Caching, Matrix Builds, and Flaky-Test Quarantine

You will take a slow, serial pipeline in this tutorial and rebuild it around three mechanisms: caching that invalidates when it should and only then, a matrix that parallelises independent work instead of multiplying it, and a quarantine lane that stops flaky tests from blocking merges without pretending they do not exist. Each mechanism has a failure mode that makes pipelines slower or less trustworthy than the naive version, and each one is covered below, because a fast pipeline nobody trusts gets bypassed, and a bypassed pipeline is decoration.

The examples use GitHub Actions syntax, but every rule translates: cache keys, matrix design, and flake policy are properties of the pipeline’s logic, not of the vendor’s YAML.

Cache keys are the whole game

A cache is a bet that a key accurately names its contents. Win the bet and you skip work safely; lose it in one direction and you rebuild unnecessarily, in the other you test stale artefacts and ship the consequences. Most CI caching disappoints because the key was chosen casually: the lockfile hash alone, or worse, a static string with a restore prefix that silently resurrects old content.

Layer your caches by how their contents change. Dependency downloads change when the lockfile changes. Compiled artefacts change when source and toolchain change. Tool installations change when the tool version changes. One cache with one key cannot express three different invalidation stories, so give each layer its own.

      - name: Restore dependency cache
        uses: actions/cache@v4
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            npm-${{ runner.os }}-
 
      - name: Restore build cache
        uses: actions/cache@v4
        with:
          path: |
            node_modules/.cache
            dist/.tsbuildinfo
          key: build-${{ runner.os }}-${{ hashFiles('package-lock.json', 'tsconfig.json') }}-${{ github.sha }}
          restore-keys: |
            build-${{ runner.os }}-${{ hashFiles('package-lock.json', 'tsconfig.json') }}-

Read the two keys closely, because the difference between them is the lesson. The dependency cache keys on the lockfile alone and restores by prefix: if the lockfile changed slightly, yesterday’s download cache is still a useful starting point, and the package manager reconciles the difference. The build cache includes the commit SHA in its primary key, so every commit writes a fresh entry, but the restore key falls back to the most recent entry for the same lockfile and compiler configuration, which is exactly what incremental build tools are designed to consume. The toolchain configuration is in the key because a compiler upgrade must not inherit artefacts from the old compiler. That single omission, the toolchain missing from the key, is responsible for a large share of “CI passed but production broke” stories involving caches.

Two operational rules finish the caching picture. First, measure hit rate and restored size per cache, and delete caches that do not pay for their restore time; restoring a huge cache over the network can cost more than the work it saves, and caches accumulate this debt quietly. Second, keep a scheduled no-cache run, weekly is a workable cadence, that builds from an empty cache. It is your canary: if the clean build fails while cached builds pass, your key is lying to you, and you want to learn that from a schedule, not from a release.

Matrix builds: parallelise dimensions, not copies

A build matrix runs a job across combinations of variables: operating systems, runtime versions, packages in a monorepo. Used well, it converts a long serial queue into parallel lanes. Used badly, it multiplies your pipeline into dozens of near-identical jobs that mostly re-verify the same thing, and the pipeline’s duration becomes the duration of its slowest, flakiest lane.

Design the matrix from the question each dimension answers. A runtime version dimension answers “does this work for every customer on a supported runtime”, which is a real question for a library and mostly theatre for a deployed service that runs exactly one runtime in production. An operating system dimension answers a real question for anything shipping binaries or filesystem-sensitive behaviour, and a poor one for a containerised service whose image is fixed. Test every dimension you can justify, and delete the ones you cannot; each cell is compute, maintenance, and another chance for an unrelated failure to block a merge.

    strategy:
      fail-fast: false
      matrix:
        package: [api, worker, web]
        include:
          - package: api
            test-command: test:integration
          - package: worker
            test-command: test:unit
          - package: web
            test-command: test:component

Two details in that fragment matter. fail-fast: false stops one lane’s failure from cancelling its siblings, so a single breakage gives you a complete picture in one run instead of a drip-feed of failures across three. And the matrix is over packages, the dimension along which the work genuinely differs, with the test command attached per cell rather than one generic command stretched across unlike suites.

The deeper technique is splitting by cost, not by category. Run the fast, high-signal checks, lint, type checking, unit tests, as a gate that must pass before the expensive lanes start. A pipeline that spends forty minutes of integration testing on code that fails a type check has its sequencing backwards. Structure the workflow so cheap certainty is bought first and expensive certainty is bought only for code that earned it.

Shard long test suites by historical duration rather than by file count. Test runners that support splitting accept timing data from previous runs and balance lanes by expected seconds, which removes the classic outcome where one lane runs three times longer than the others because it drew the slow files. If your runner cannot do this, split by directory and rebalance by hand whenever the imbalance becomes visible in the timings you are already collecting.

Flaky tests: quarantine is a policy, not a delete key

A flaky test passes and fails on identical code. It is worse than no test, because it trains the team to re-run pipelines until they go green, and a team trained to ignore red has no alarm system at all. The instinct to delete flaky tests is understandable and wrong: the test often covers behaviour that matters, and its flakiness is information about the test’s design, occasionally about the product’s timing assumptions.

Quarantine is the middle path, and it only works with rules that have teeth:

1. Detection is automatic. A test that fails and then passes on retry, or that fails intermittently across runs on unchanged code, is flagged by tooling, not by someone’s memory. Most CI systems can surface rerun-passed tests; treat that report as the quarantine queue.
2. Quarantined tests still run. They move to a separate lane that does not block merges, with results visible and counted. A quarantined test that is not running is a deleted test with extra steps.
3. Quarantine expires. Each quarantined test carries an owner and a deadline. At the deadline it is fixed, rewritten, or deleted by a human decision, never silently renewed. An expiry is what separates a quarantine from a graveyard.
4. The budget is small and public. A team that agrees, for example, that quarantined tests may not exceed a small fixed count has a forcing function: a new flake means something gets fixed this week. Publish the count where the team plans work.

  quarantine:
    name: Flaky-test lane (non-blocking)
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - name: Run quarantined suite
        run: npm run test:quarantine
      - name: Report flake results
        if: always()
        run: node scripts/report-flakes.js

Note what the lane does beyond running the tests: it reports, on every run, whether quarantined tests passed or failed, feeding the data that decides whether a test has earned its way back to the blocking suite. Promotion back should be earned by a sustained run of passes, and the criteria for it belong in the same document as the quarantine rules.

Retries deserve one sentence of their own: automatic retry of a failed test inside the blocking lane hides flakes and converts them into slow, mysterious merges. Retry in the quarantine lane if you must; in the blocking lane, a failure should fail.

What to measure so this stays true

Pipelines decay, so instrument yours like a service. Track total duration and queue time separately, because a fast job behind a long queue is still a slow pipeline and the fixes differ. Track per-lane duration to catch matrix imbalance. Track cache hit rates beside the durations, since a change in either without the other is a clue. And track the rerun rate, how often a developer re-runs a failed pipeline without a code change, because that number is your trust metric: when it climbs, the team has stopped believing red means broken, and no amount of caching will buy that back.

Verdict

Cache in layers, with keys that name everything the contents depend on, including the toolchain, and prove the keys honest with a scheduled clean build. Build matrices along the dimensions where behaviour genuinely differs, gate the expensive lanes behind cheap checks, and shard by measured duration. Quarantine flaky tests automatically, keep them running in a visible non-blocking lane, and give every quarantine an owner and an expiry. Do those three things and CI becomes what it was supposed to be: fast because it skips work provably, and trusted because red still means stop.