Microservices multiply the CI/CD problem. One pipeline becomes fifty. Each service has its own build, test, and deploy lifecycle. Without a coherent strategy, you get either a centralized bottleneck or fifty snowflake pipelines that nobody understands.
The Pipeline HLD
A microservices CI/CD system has four stages:
Code Push -> Build -> Test -> Deploy -> Verify
↓ ↓ ↓ ↓ ↓
Trigger Artifact Gates Strategy Monitor
Each stage has decisions that affect speed, safety, and complexity.
Build: One Artifact Per Commit
Build once, deploy many times. The CI pipeline produces an immutable artifact (container image) tagged with the commit SHA. That exact artifact promotes through environments. Never rebuild for different environments.
# GitHub Actions example (simplified)
- name: Build and push
run: |
docker build -t myapp:${{ github.sha }} .
docker push myapp:${{ github.sha }}
Environment-specific configuration comes from external sources (ConfigMaps, parameter stores), not from rebuilding the image.
Test: Gates That Matter
Not all tests belong in the pipeline. The goal is fast feedback on common failures, not exhaustive verification.
Run in pipeline (blocking):
- Unit tests (fast, < 5 minutes)
- Linting and static analysis
- Container image scanning (vulnerabilities)
- Contract tests (if using consumer-driven contracts)
Run async (non-blocking):
- Integration tests (slower, may be flaky)
- Performance tests (nightly or on-demand)
- End-to-end tests (valuable but slow and brittle)
If the pipeline takes more than 15 minutes, developers will bypass it or batch changes. Speed matters more than comprehensiveness for the blocking gates.
Deploy: Progressive Delivery
Direct to production is fine for low-risk services with good monitoring. For everything else, use progressive delivery:
- Deploy to staging: Automated tests, manual verification if needed
- Canary to production: 5% traffic, monitor error rates
- Gradual rollout: 25%, 50%, 100% with pauses for verification
- Automatic rollback: If metrics degrade, revert without human intervention
Tools: Argo Rollouts, Flagger, or native Kubernetes rolling updates for simpler cases.
Trunk-Based Development vs GitFlow
Trunk-based: Developers commit directly to main (or via short-lived feature branches merged within a day). Every commit to main triggers the pipeline. Feature flags control incomplete work.
GitFlow: Long-lived develop, release, and feature branches. Complex merge patterns. Releases are cut from release branches.
For microservices, trunk-based wins. GitFlow's complexity multiplies across 50 services. Trunk-based with feature flags is simpler and enables continuous delivery.
The objection is always "but we need to control what goes to production." Feature flags solve this better than branches. A flag can be toggled in production without a deploy. A branch requires a merge, build, and deploy to change behavior.
# Feature flag example
if feature_flags.is_enabled("new-checkout-flow", user_id):
return new_checkout(user_id)
else:
return legacy_checkout(user_id)
Pipeline Per Service vs Shared Pipeline
Option A: Each service has its own pipeline definition. Maximum flexibility. Each team customizes as needed. Risk: 50 different pipelines with inconsistent security, testing, and deployment practices.
Option B: Shared pipeline template. Platform team maintains a standard pipeline. Services opt in with minimal configuration. Less flexibility, but consistent and maintainable.
The pragmatic approach is B with escape hatches. Provide a standard pipeline that handles 80% of services. Allow customization for the 20% with special needs (via pipeline parameters, not forks).
# Shared pipeline template (simplified)
# Service repos reference this, providing only service-specific values
parameters:
serviceName: "my-api"
testCommand: "npm test"
deployStrategy: "canary"
DORA Metrics: What to Track
DORA (DevOps Research and Assessment) defines four key metrics:
- Deployment frequency: How often you deploy to production. Elite: multiple times per day.
- Lead time for changes: Time from commit to production. Elite: under 1 hour.
- Change failure rate: Percentage of deployments causing incidents. Elite: under 15%.
- Time to restore: How quickly you recover from failures. Elite: under 1 hour.
Track these per team, not just org-wide. Averages hide struggling teams.
What DORA does not measure: Code quality, developer satisfaction, business impact. Use DORA as a health indicator, not the sole success metric.
Monorepo vs Polyrepo
Monorepo (one repo for all services):
- Atomic changes across services
- Shared tooling and configuration
- Better code discoverability
- Scaling challenges (build times, access control)
Polyrepo (one repo per service):
- Clear ownership boundaries
- Independent versioning
- Simpler CI configuration per repo
- Harder to make cross-service changes
For microservices, polyrepo is more common and scales better organizationally. Monorepo works well for smaller teams (< 50 engineers) or with tools like Bazel that handle large repos efficiently.
The CI implication: monorepo needs path-based triggers (only build changed services). Polyrepo gets this naturally (each repo has its own pipeline).
Secrets in Pipelines
CI pipelines need secrets (registry credentials, deployment tokens, API keys). Never hardcode them.
- Use the CI system's secret store (GitHub Secrets, GitLab CI variables)
- Prefer OIDC federation over long-lived tokens (GitHub Actions can assume AWS IAM roles without stored credentials)
- Scope secrets to the minimum necessary (per-environment, per-service)
- Rotate regularly and audit access
Speed Optimizations
If pipelines are slow:
- Parallelize tests: Split test suites across multiple runners
- Cache dependencies: Don't reinstall node_modules on every run
- Use smaller base images: Alpine or distroless instead of full OS images
- Skip unnecessary steps: Don't run e2e tests on documentation changes (use path filters)
- Buildkit caching: Use Docker layer caching aggressively
Target: under 10 minutes from push to staging. Under 15 minutes to production with canary.
Tradeoff Deep-Dive: Speed, Safety, and the Pipeline Tax
Every pipeline decision trades speed against safety, and the exchange rate changes with team size. For five engineers, a 20-minute pipeline with exhaustive end-to-end tests is fine; deploys are infrequent and everyone knows the system. For fifty engineers merging to trunk all day, that same 20 minutes becomes a queue: developers stack changes waiting for green builds, batch sizes grow, and each deploy carries more risk, which is exactly what the long pipeline was trying to prevent. The pipeline tax is regressive: slow pipelines hurt large teams disproportionately.
The standard resolution is tiered gates: fast blocking checks (unit tests, lint, scan, build under 10 minutes) and slow async verification (integration, performance, end-to-end) that does not block the merge but does block production promotion. This works only if the async results are actually enforced somewhere. The failure mode every team hits: async tests go red, nobody looks, and production promotion happens anyway because the check was advisory. Make the promotion gate read the async results. A pipeline stage that nobody enforces is documentation, not a gate.
Canary analysis has its own tradeoff that interviews probe: automatic rollback on metric degradation against false positives. Set the canary sensitivity too high and every deploy rolls back on noise; too low and real regressions sail through. The mature setup pairs automated analysis (error rate, latency, saturation compared against the stable baseline) with a human-readable reason for every automated decision, logged where the deploying engineer can see it. A canary system that rolls back without explaining why will be disabled within a quarter; a canary system that explains itself gets trusted.
The monorepo versus polyrepo choice resurfaces here as a CI scaling question. Polyrepo pipelines are simple and independent but multiply fixed costs: fifty repos means fifty pipeline definitions drifting apart unless a shared template enforces consistency. Monorepo pipelines need path-based triggers and affected-service computation, which is genuinely complex (Bazel or Nx earn their keep here), but give atomic cross-service changes and one place to enforce standards. The deciding factor is rarely technical: it is whether the organization can maintain template discipline across repos. If not, the monorepo's enforced consistency wins despite the tooling cost.
System Architecture
+--------------+ +------------------+
| Pull request|---->| CI build test |
| trunk | | scan package |
+--------------+ +--------+---------+
|
v
+------------------+
| Artifact |
| registry SHA |
+--------+---------+
|
v
+------------------+
| CD progressive |
| delivery |
+--------+---------+
|
+---------+---------+
| |
v v
+------------------+ +------------------+
| Staging | | Production |
| full rollout | | canary steps |
+------------------+ +------------------+
A pull request against trunk triggers CI: build, test, scan, package. The immutable artifact, tagged by commit SHA, lands in the registry. CD promotes that exact artifact: full rollout to staging, then canary steps in production with automated analysis and rollback. Feature flags sit alongside this flow, decoupling release (code deployed) from launch (behavior enabled), so a deploy and a release are independent decisions. Pipeline events feed DORA metrics per team.
Scalability
At 10x services, CI runner queue depth grows, pipeline minutes become a real budget line, artifact registry storage and pull throughput limits appear, and the GitOps repo becomes a commit-contention hotspot as every service's CD updates image tags.
At 100x, the pain points shift: shared pipeline template versioning (v1 versus v2 across hundreds of repos, with migration); test grid flakiness at scale (a 0.1% flake rate means constant red builds across a thousand daily runs); canary analysis metrics cardinality exploding in the observability backend; deployment serialization per environment (one production deploy at a time per service, queued).
The scaling strategy: ephemeral autoscaled runner pools (scale to zero, fast start); path-based triggers so documentation changes skip the full pipeline; test sharding across runners with flaky-test quarantine (quarantine, do not delete; track and fix); registry lifecycle policies deleting old SHAs; per-service deploy queues with concurrency limits; parallel canary stages where services are independent; aggressive dependency caching, which is the single highest-ROI pipeline optimization.
Security Considerations
Identity: OIDC federation from the CI system to the cloud (GitHub Actions assuming IAM roles) instead of long-lived access keys; per-environment deploy roles with least privilege (the staging role cannot touch production). Artifact integrity: sign images with cosign, generate SLSA provenance, verify signatures at deploy time; image scanning as a blocking gate for critical CVEs, advisory for the rest. Pipeline hygiene: branch protection with required reviews; never run privileged pipeline steps on code from fork PRs (build in an untrusted context, promote only after merge); secret masking in logs; per-service scoped secrets.
Realistic attack vectors: a compromised dependency executing code during the build with access to deployment credentials (mitigate with OIDC short-lived tokens scoped to the single job, and egress restrictions on build runners); a malicious PR exfiltrating secrets through build logs or test output (mitigate: fork PRs get no secrets at all); an overly broad deploy role letting any branch deploy to production (mitigate: role assumption restricted to the production branch and the CD identity, never developer identities); unsigned artifacts deployed through a compromised registry credential (mitigate: signature verification as an admission check, with registry write credentials separated from deploy read credentials).
Production Checklist
Signals: pipeline p95 duration and queue time per stage; pipeline failure rate per stage; deployment frequency, lead time, change failure rate, time to restore (DORA, tracked per team); canary rollback rate; artifact signing verification failures.
Alert thresholds: pipeline failure rate spiking above 3x baseline over an hour; runner queue depth above threshold for 15 minutes (a capacity problem); canary error-budget burn rate; any signature verification failure; change failure rate trending above 15%.
Failure modes seen in practice: runner pool exhaustion blocking all deploys company-wide (runbook: priority queue for production hotfixes, pre-provisioned standby capacity on the CD-critical path); a flaky end-to-end suite blocking trunk for hours (runbook: quarantine the flaky test, merge, fix forward; never let one test hold the trunk hostage); a bad canary metric causing false rollbacks on every deploy (runbook: disable automated analysis, fall back to manual promotion, fix the metric); a registry outage blocking image pulls (runbook: multi-region registry replication, or at minimum a warm cache of recent images on nodes); a feature flag left enabled after a partial rollout causing an incident days later (runbook: flag ownership and expiry dates, audit flags older than 30 days).
Runbook notes: one-click rollback to the previous SHA is the most important pipeline feature; the flag kill switch is the second. Document the deploy-freeze procedure (who can declare it, how it is communicated, how it is lifted). Rehearse rollback, not just rollout.
The best CI/CD pipeline is boring. It runs the same way every time, fails loudly when something is wrong, and gets out of the developer's way.