Software Testing in CI/CD Pipelines

A CI/CD pipeline treats the test suite as a quality gate. Every time code is pushed, the pipeline runs tests and either lets the change through or blocks it.

For this to work well, tests must be:

Fast — a 60-minute test suite blocks developers for an hour per commit. Target under 10 minutes for the main feedback loop; run slower tests in parallel or on a separate schedule.

Deterministic — a test must always produce the same result on the same code. Non-deterministic (flaky) tests undermine the trustworthiness of the gate.

Independent — tests must not rely on execution order. Any test should be able to run in any position without affecting others.

Complete — the suite should cover the code changes being merged, particularly the critical paths.

Flaky tests — tests that randomly pass or fail create noise and erode trust. Developers begin ignoring failures, masking real bugs. Research suggests that 14–16% of test failures in large suites are attributable to test flakiness.

Slow builds — as suites grow, build times creep up. When a full run takes 45 minutes, developers push fewer commits, batch changes, and accumulate more risk per deployment.

Insufficient parallelisation — running tests sequentially when they could run in parallel is the most common cause of unnecessarily slow pipelines.

Poor test granularity — end-to-end tests are slow, brittle, and expensive to maintain. A healthy test suite has many fast unit tests, fewer integration tests, and very few E2E tests (the testing pyramid).

Missing test environments — tests that only run locally, not in CI, miss environment-specific failures. All tests that matter should run in CI.

The testing pyramid — the classical model divides tests into three layers: many unit tests (fast, cheap), fewer integration tests (medium), and very few end-to-end tests (slow, expensive). CI/CD pipelines should reflect this pyramid.

Test splitting — divide the suite into fast (unit) and slow (integration/E2E) groups. Run fast tests on every push. Run slow tests on merge to main, or on a scheduled basis.

Parallelisation — use CI platforms' built-in parallelisation to distribute tests across multiple workers. Combine with test prioritization to ensure each worker is loaded evenly.

Caching — cache dependencies and build artefacts between runs. Most CI platforms support this natively.

Quarantine — move known flaky tests to a separate, non-blocking suite while they are being fixed. Do not let known flaky tests block production deployments.

Track these metrics to understand the health of your testing pipeline:

Pass rate — what percentage of builds pass on the first run? Low pass rates indicate either real bugs or high flakiness.

Flake rate — what percentage of test failures are not reproducible on retry? High flake rates indicate systemic unreliability.

Build time — how long does the full pipeline take? Track over time to catch regressions.

Test coverage — what percentage of the codebase is covered by tests? Track critical paths specifically.

Time to feedback — how long does a developer wait to know if their change is good? This is the metric that most directly affects developer productivity.