Hardening the pipeline should not slow delivery. These light-touch controls catch what we and our contractors and partners see most—without breaking developer flow.
This guide explains the minimum guardrails to protect your build and release process from common CI/CD attack paths, how to roll them out safely, and how to measure that they stay in place.
Key definitions (plain language)
Runner/Executor: The machine, container, or service that executes your jobs. Treat it like production: it can read code, secrets, and build artifacts.
Artifact: Anything produced by your build (packages, images, binaries, manifests). Artifacts move between stages and must be trusted explicitly.
Provenance: Evidence that an artifact was built by trusted code on trusted infrastructure with known inputs.
Brokered secrets: Short-lived credentials issued at job start, scoped to the minimum the job needs, and revoked on completion.
Why executives should care
- • CI/CD is the narrow bridge between source code and customers. If attackers control it, they can ship their code to your users.
- • Guardrails reduce blast radius and create proofs: you can show auditors and customers that only verified artifacts reach production.
- • Lightweight controls avoid the usual tradeoff: security signals go up while delivery speed stays high.
Scope, ownership, and change control
We and our contractors and partners start with a written scope: which repos, environments, and pipelines are in-bounds and who approves changes to sensitive jobs. We name owners for runner fleets, secret stores, and release steps so accountability is clear.
- • Define protected paths (e.g., release pipelines, infra-as-code) that require code review and signed changes.
- • Document who can create or approve runners and what images they are allowed to use.
- • Require two-person review for changes that touch deploy steps, signing keys, or secret scopes.
Safe handling: keys and secrets in CI/CD
We and our contractors and partners separate long-lived root keys from day-to-day pipeline use. Build and signing keys live in an HSM or cloud KMS; jobs receive short-lived, least-privilege tokens issued at runtime and logged for audit.
- • No plaintext long-lived secrets in repos, job variables, or runner images.
- • Use OIDC or workload identity to mint per-job tokens with explicit audiences and expiry in minutes.
- • Rotate client secrets on role changes and incident response; measure mean time to revoke.
Baseline controls
- • Scoped runners/executors: isolate by repo/tenant; avoid shared privileged runners.
- • Artifact signing + verification before promotion/deploy; fail closed on mismatch.
- • Short-lived secrets via brokers; keep prod secrets out of generic pipelines.
- • Dependency provenance: lockfiles and internal mirrors for critical paths.
Method: minimal guardrails, maximum leverage
We and our contractors and partners start with one protected pipeline and one service. We enable signed artifacts and verification at deploy, move deploy credentials behind a broker, and migrate that pattern across repos. Small, repeatable steps beat big-bang rewrites.
- • Protect the release pipeline first, then expand to build and test stages.
- • Pin base images and runners; rebuild regularly from code, not by hand.
- • Add verification early—promotion gates should check signatures and policy before any deploy happens.
What you get (and what you won’t)
- • Replayable steps: exact YAML/job diffs and CLI commands to enable signing, verification, and brokered secrets.
- • Reference runner images and IaC snippets you can adopt with minimal editing.
- • Guardrail checklist mapped to owners: who runs what and how it is monitored.
- • What you won’t get: a theoretical lecture on supply chain risk without deployable changes.
Anatomy of a CI/CD compromise
- Foothold: attacker gains access to a shared runner or a repo with pipeline permissions.
- Secrets: static tokens in job variables or logs leak access to registries or cloud roles.
- Artifact swap: unsigned or unverified artifacts are replaced between build and deploy.
- Lateral movement: pipeline writes to production environments beyond intended scope.
- Impact: malicious code shipped to users or control over infrastructure.
Metrics leaders can track
- • Runner isolation coverage: Percent of jobs on repo/tenant-scoped runners (not shared privileged pools).
- • Signed artifact rate: Percent of deploys that verify signatures and provenance before release.
- • Secret lifetime: Median credential TTL issued to jobs; aim for minutes, not days.
- • Base image freshness: Days since last runner/base image rebuild from code.
- • Blocked promotion attempts: Number of deploys refused by verification gates (a healthy early signal).
Best practices: runners and executors
- • Per-repo or per-tenant runners with no shared admin pools; disable Docker-in-Docker unless required.
- • Immutable runner images built from code; rebuild on a schedule and after critical CVEs.
- • Network egress policies on runners; no default access to production networks.
Best practices: artifacts and releases
- • Sign at build using keys in KMS/HSM; store attestations with artifacts.
- • Verify at promotion/deploy; fail closed and alert on mismatch or missing attestation.
- • Keep SBOMs for critical services; diff on changes to sensitive components.
Best practices: secrets and identity
- • Use workload identity (e.g., OIDC) to mint per-job credentials; scope by repo, branch, and job purpose.
- • Remove long-lived cloud keys from job variables and logs; scrub outputs by default.
- • Rotate secrets automatically; alert on stale tokens or unused access.
Best practices: dependencies and inputs
- • Pin versions via lockfiles; review changes to transitive deps on sensitive services.
- • Use internal mirrors for critical packages and container bases; alert on unexpected sources.
- • Scan before build, not after release; fail builds on policy violations that matter.
Common anti-patterns that backfire
- • Shared privileged runners for everything (one compromise, total blast radius).
- • Signing without verification gates (no one checks the signature at deploy).
- • Sprinkling static secrets across jobs; they leak in logs and cache layers.
- • Treating the pipeline as a dev toy instead of a production system.
Delivery, training, and handoff
We and our contractors and partners deliver runnable examples for your CI provider, plus training for platform and service teams. We document who owns keys, runner images, and verification policies, with playbooks for rotation and incident response.
How to roll out safely
- Pick one critical service and protect its release pipeline end-to-end (runners → signing → verification → brokered secrets).
- Template the working config; apply to the next two services with small adjustments.
- Enable alerts and dashboards for verification failures and secret issuance.
- Schedule quarterly key rotations and runner rebuilds; verify with a dry run.
- Expand to broader repos once owners are comfortable and metrics stabilize.
What success looks like over 6–12 months
- • Most deploys blocked on verification failure during early rollout, then a steady decline as patterns stick.
- • No shared privileged runners on critical paths.
- • All production artifacts signed and verified at promotion/deploy.
- • Secrets issued per job with short lifetimes and clear audit trails.
CI/CD guardrails should feel boring and durable. Done well, they turn scary supply-chain headlines into simple checks that run every day. We and our contractors and partners help you pick the few controls that matter, wire them into your pipelines, and keep them there without slowing teams down.