CI/CD Best Practices

CI/CD is the system that turns code into production. Bad CI/CD means slow feedback, manual deployments, and fear of shipping. Good CI/CD means commits get tested instantly, problems surface immediately, and deployments happen safely multiple times a day.

Too many teams have "CI/CD" in name only. They run some tests in a GitHub Actions workflow, maybe do some manual steps, maybe wait for an approval email. That's not CI/CD. That's automation theater.

Real CI/CD is a feedback loop: developer commits → tests run → feedback arrives in minutes → problems caught immediately. Every commit that passes tests is deployable. Every commit that breaks something alerts the team immediately.

Why This Matters

Speed of feedback is everything. A developer finishes a feature at 3pm. If CI runs in five minutes, they know it's good or broken while they're still in the context of what they wrote. If CI takes an hour, they're onto the next thing. When tests fail, they're fighting to remember what they did.

Safety is a side effect of good CI/CD. When you deploy multiple times a day with automated tests and monitoring, each deploy is small. Small deployments are low-risk. If something breaks, you fix or rollback a small change. If deployments are quarterly, you're deploying a million lines of changes. That's dangerous.

Confidence scales. A team that deploys once a quarter is terrified of deploying. A team that deploys hourly treats it as routine. The routine teams have fewer outages.

CI Fundamentals

Trunk-based development is the foundation. Everyone commits to main (or trunk). Feature branches are short-lived—hours or a day, not weeks. When people work on different branches for weeks, integrating becomes painful. When people integrate continuously, small conflicts get resolved immediately.

Long feature branches delay integration and make large merge conflicts. They also hide problems—your code works on your branch but breaks when integrated. You don't find out until days later when the merge happens.

The alternative, GitHub flow, is similar: create a branch, merge quickly. Branches exist to avoid blocking others, not to avoid integration.

Every commit gets tested. A developer pushes a commit. Within minutes, tests run. If tests fail, the developer fixes it. If tests pass, the code is eligible for release. This pairs with test-driven development — when tests are written first, the pipeline has meaningful coverage from day one. There's no separate "integration" phase where things mysteriously break.

The build must be green. When the main branch is broken, the entire team stops. They fix it. They don't commit on top of broken code. This discipline prevents cascading failures and keeps feedback tight. A broken build is the highest priority interrupt.

Fast feedback is non-negotiable. A CI pipeline that takes an hour defeats the purpose. You want results in five to fifteen minutes. If tests are slow, parallelize them. If builds are slow, cache dependencies. If there's one test that takes ten minutes, isolate it and run it separately.

Pipeline Design

A CI pipeline has stages. Each stage is a gate.

Stage 1: Lint and format. Run linters, format checkers, security scanners. This happens first because it's fast. If code doesn't meet style requirements or has obvious security issues, fail immediately.

lint:
  script:
    - eslint src/
    - prettier --check src/
    - npm audit
  cache:
    paths:
      - node_modules/

Stage 2: Unit tests. Run unit tests in parallel. These are fast and catch most bugs.

test_unit:
  script:
    - npm run test:unit
  parallel:
    matrix:
      - SHARD: [1/4, 2/4, 3/4, 4/4]

Stage 3: Build. Compile or bundle your code. This step catches syntax errors and import issues.

build:
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

Stage 4: Integration tests. Run tests that exercise multiple components. These are slower but catch integration bugs.

test_integration:
  script:
    - docker-compose up -d
    - npm run test:integration
    - docker-compose down

Stage 5: Deploy to staging. Deploy to a production-like environment. Run smoke tests.

deploy_staging:
  script:
    - deploy-to-environment.sh staging
  environment: staging

Stage 6: E2E tests on staging. If critical, run a few end-to-end tests against the staging environment.

Stage 7: Deploy to production. If all gates pass, deploy to production. In mature teams, this is automatic. In others, it requires manual approval.

deploy_production:
  script:
    - deploy-to-environment.sh production
  environment: production
  when: manual

Caching and Parallelism

A slow pipeline kills velocity. Invest in making it fast.

Cache dependencies. Don't download npm packages on every run. Cache them between runs. Only invalidate the cache when dependencies change.

cache:
  paths:
    - node_modules/
  key: $CI_COMMIT_HASH_of_package-lock.json

Parallelize tests. If you have 1000 unit tests, split them across machines. One machine runs 250, another 250, etc. Run them in parallel.

Use Docker layer caching. If you're building Docker images, layer them intelligently so unchanged layers get cached.

FROM node:18
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

Copying package files before application code means the install layer caches even if application code changes.

Deployment Patterns

Blue-green deployment runs two identical environments. Deploy new code to the green environment while blue is live. Test green thoroughly. When confident, switch traffic to green. If something breaks, switch back to blue immediately.

Blue (live) ← load balancer → Green (new code)
If green is healthy:
  Blue becomes green
  Green becomes blue (idle, ready to test new code)

This is the safest pattern because rollback is instant. It requires the infrastructure to support two environments, which most cloud platforms do easily.

Canary deployment sends a small percentage of traffic to new code. If metrics stay healthy, gradually increase traffic. If metrics degrade, rollback immediately.

Canary is safer than all-or-nothing but requires infrastructure that can split traffic.

Feature flags let you deploy code without activating it. New code ships to production but only executes for certain users.

if (flags.isEnabled('new_checkout_flow')) {
  return checkoutFlowV2(user);
} else {
  return checkoutFlowV1(user);
}

This lets you deploy safely and test with real traffic before enabling broadly.

Common Mistakes

Slow pipelines. A 45-minute pipeline defeats the purpose of CI. Invest in speed. Profile your pipeline. Find bottlenecks. Parallelize. Cache. Your pipeline should run in under fifteen minutes.

Flaky tests. Tests that fail intermittently destroy confidence. Developers ignore failures. Eventually the pipeline becomes useless. Fix flaky tests immediately or remove them.

Skipping stages. Developers push code that doesn't run through the full pipeline because "it's just a config change" or "it's too slow." This is how problems sneak through. Every commit goes through the full pipeline.

No monitoring of deployed code. You deployed. Great. Now what? You need monitoring, alerting, and logs. If the code breaks in production, you should know within minutes, not hours. Combine this with error handling and resilience patterns so your system degrades gracefully when issues slip through.

Manual deployment steps. Manual steps are error-prone. Automate everything. If you're logging into a server and running commands, automate it.

Not testing the deployment process. Your deployment script should be tested. Not in production. Test it in staging every time.

CI/CD in AI-Assisted Development

When agents generate code, the pipeline becomes even more critical. Agents might generate a lot of code. All of it needs to pass through the same gates. A well-designed pipeline catches issues in generated code early.

Agents can also help maintain CI/CD. They can write tests for generated code, update configuration, and fix pipeline failures. The clearer the pipeline definition, the better agents understand and work with it.

FAQ

Primary Sources

• Nicole Forsgren's research-backed guide to measuring and improving software delivery. Accelerate
• Jez Humble and David Farley's definitive guide to continuous delivery and deployment. Continuous Delivery
• The DevOps Handbook's comprehensive guide to building high-performing organizations. DevOps Handbook
• Google's engineering practices documentation on code quality and deployment safety. Google Eng Practices
• The Pragmatic Programmer's approach to sustainable development practices. Pragmatic Programmer
• Google SRE team's guide to reliability engineering and operational excellence. SRE Workbook
• Robert Martin's handbook on clean code and sustainable software design. Clean Code