When I started handling production deployments about a decade ago, the process was often just SSH into a server, pull the latest code, and restart the service while hoping nobody noticed the downtime. If something broke, you scrambled to revert. Modern deployment strategies exist to remove that anxiety. For teams running on Kubernetes, VMs, or even serverless platforms, the choice between rolling updates, blue-green, and canary deployments determines how much risk you expose to users, how quickly you can recover, and how much infrastructure you need to pay for. In this article, I will break down how each strategy actually works under the hood, what I have seen go right and wrong with each one in real environments, and how to pick the right approach for your team and workload. This is squarely part of your GitHub Actions Advanced Workflows: Matrix Builds, Reusable Actions and OIDC thinking about CI/CD Pipelines, because your pipeline does not end when the artifact is built — it ends when users are safely running the new version.
Rolling Updates: The Default Workhorse Behind Kubernetes Deployments
Rolling updates are the strategy most teams use first, often without realizing it, because it is the default for Kubernetes Deployments. The idea is simple: instead of stopping all old instances at once, you gradually replace them with new ones. If you have 10 pods running v1, the controller will start one or two pods with v2, wait until they are ready, then terminate one or two v1 pods, and repeat until all pods are on v2. Users never see a full outage, and you do not need double the infrastructure.
In my experience, rolling updates are ideal for stateless web services and APIs where backward compatibility is maintained. I have run rolling updates for everything from internal dashboards with 3 replicas to customer-facing APIs with 80 pods. They are predictable and cheap. Where teams get into trouble is assuming they are zero-downtime by default. They are not, unless you have configured readiness probes, graceful shutdown, and proper PodDisruptionBudgets.
How a Rolling Update Behaves in Kubernetes
The behavior is controlled by two fields: maxUnavailable and maxSurge. maxUnavailable defines how many pods can be unavailable during the update, while maxSurge defines how many extra pods can be created above the desired replica count. For a production service where availability matters, I typically set maxUnavailable: 0 and maxSurge: 1 or 25%. This ensures you always have capacity while the rollout proceeds. If you set maxUnavailable: 1 with 3 replicas, you can briefly run with only 2 pods, which can cause latency spikes if you are already near capacity.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
labels:
app: checkout-api
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 2
selector:
matchLabels:
app: checkout-api
template:
metadata:
labels:
app: checkout-api
version: v2.1.0
spec:
containers:
- name: api
image: registry.example.com/checkout-api:v2.1.0
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
Notice the readinessProbe and preStop hook. Without a readiness probe, Kubernetes will send traffic to a pod that is still starting, resulting in 502s. Without a graceful termination period, in-flight requests get killed when the old pod is terminated. I have seen both cause more incidents than the actual code change. According to the Kubernetes Documentation, a Deployment ensures only pods with a passing readiness check receive traffic, which is why defining that check is not optional.
When Rolling Updates Fall Short
Rolling updates run both versions simultaneously for a short window. That means v1 and v2 must be able to run against the same database schema and shared dependencies. If v2 introduces a breaking schema migration, you will have v1 pods writing data in the old format while v2 pods expect the new format. I have found that the safest approach is to make database changes backward compatible and deploy them in two phases: first, a change that both versions understand, then the code change. If you cannot guarantee compatibility, rolling updates are not the right tool.
Rollbacks are also not instantaneous. Kubernetes can roll back to the previous ReplicaSet, but it is still a rolling process. If v2 is causing data corruption or high error rates, you will still have a few minutes where bad pods are serving traffic while the rollback completes.
Blue-Green Deployments: Two Identical Environments and a Single Router Switch
Blue-green takes a completely different approach. You run two full environments — blue (current live) and green (new version) — and switch all traffic from one to the other at once. The unused environment remains idle, ready to become live again if you need to revert.
I first used blue-green for an e-commerce platform where even 30 seconds of inconsistent behavior during a rolling update was unacceptable. We provisioned two identical Auto Scaling Groups behind an Application Load Balancer. Blue was serving 100% of traffic. We deployed green, ran smoke tests and synthetic transactions against it, and then flipped the target group in the load balancer. The cutover took seconds. When we found a critical tax calculation bug ten minutes later, we flipped back to blue just as fast. No rolling, no waiting.
The operational clarity is the main advantage. At any moment, you know exactly which version is serving traffic. Testing is more realistic because you can run full integration tests against the green environment with production-like data and configuration before anyone sees it. For compliance-heavy workloads or major version jumps, that isolation is valuable.
Implementing Blue-Green Without Doubling Everything Forever
The obvious downside is cost. You need twice the compute, at least temporarily. For Kubernetes, the pattern is often implemented by running two Deployments and switching a Service selector, or by using a deployment controller like Argo Rollouts.
# Two deployments: checkout-api-blue and checkout-api-green
# Service initially points to blue
apiVersion: v1
kind: Service
metadata:
name: checkout-api-active
spec:
selector:
app: checkout-api
slot: blue # flip to 'green' to switch traffic
ports:
- port: 80
targetPort: 8080
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout-api
spec:
replicas: 6
strategy:
blueGreen:
activeService: checkout-api-active
previewService: checkout-api-preview
autoPromotionEnabled: false
scaleDownDelaySeconds: 600
In this Argo Rollouts example, autoPromotionEnabled: false means the green rollout will not automatically receive production traffic until you manually promote it. The scaleDownDelaySeconds: 600 keeps the old blue ReplicaSet running for ten minutes after promotion, giving you a quick rollback window. This is a pattern I have relied on for high-stakes releases where I want a human to verify metrics before committing.
You do not always need to double your database. In most cases, blue and green share the same database and cache layers. The environments that are duplicated are stateless compute. You should still plan for database backward compatibility, but the blast radius for code bugs is smaller because you never have mixed versions serving at the same time. If your artifact pipeline is not already versioning and storing immutable images reliably, fix that first by looking at Artifact Management: Container Registries, Helm Charts and Build Caching — blue-green only works when you can deterministically deploy any previous version.
Canary Releases: Steering a Controlled Percentage of Traffic to the New Version
Canary is the most surgical of the three. Instead of replacing all pods or flipping all traffic at once, you send a small percentage of real users to the new version — 1%, 5%, 10% — and watch what happens. If error rates rise or latency degrades, you can halt and roll back before most users are affected. If metrics look good, you gradually increase the percentage to 25%, 50%, then 100%.
The name comes from the canary in a coal mine. In practice, I have found canaries most valuable for large user bases and risky changes: a new recommendation algorithm, a rewritten checkout flow, or a library upgrade that could affect performance at scale. I once rolled out a change to session handling with a 5% canary. Our synthetic tests passed, but at 5% we saw a 0.8% increase in 500 errors that only appeared with real user cookie patterns. We stopped the rollout, fixed the bug, and no one outside that 5% ever noticed. A rolling update would have taken that error rate to 100% within minutes.
Traffic Shaping and Canary Analysis
Canary requires a mechanism to split traffic. In simple setups, you can run two Deployments with different replica counts — for example, 1 pod of v2 and 19 pods of v1 gives you roughly 5% traffic if your load balancer does round-robin — but that is imprecise and does not allow header-based routing. For real control, you need a service mesh or ingress controller: Istio, Linkerd, NGINX Ingress, AWS App Mesh, or Flagger/Argo Rollouts.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: checkout-api
spec:
hosts:
- checkout.example.com
http:
- route:
- destination:
host: checkout-api
subset: v1
weight: 90
- destination:
host: checkout-api
subset: v2
weight: 10
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: checkout-api
spec:
host: checkout-api
subsets:
- name: v1
labels:
version: v1.9.3
- name: v2
labels:
version: v2.0.0
With this Istio configuration, 10% of requests go to v2 regardless of how many pods each version has. You can also route based on headers — for example, sending internal testers or users with a x-canary: true header to v2 first. I have used header-based canaries to let QA and product managers validate a feature in production before rolling it to a percentage of real users.
The hard part is not splitting traffic, it is deciding when to proceed. A mature canary setup pairs traffic splitting with automated analysis. Tools like Flagger or Argo Rollouts can query Prometheus or Datadog every minute and check service-level objectives: p95 latency under 300ms, error rate under 1%, CPU under 80%. If those thresholds fail, the canary is automatically rolled back. If they pass for a configured interval, the weight is increased. Without automated analysis, I have seen teams run a 5% canary and stare at Grafana dashboards for an hour, which is both stressful and error-prone.
Canaries also work exceptionally well with feature flags. Instead of deploying two versions of the entire service, you can deploy one version with a flag that enables the new code path for a percentage of users. If you want to decouple deployment from release, read Feature Flags for Progressive Delivery: LaunchDarkly, Unleash and OpenFeature — the combination of canary infrastructure plus flags gives you the most granular control, though it adds SDK and operational overhead.
Infrastructure Cost, Risk Profile and Operational Complexity Side by Side
Choosing a strategy is a trade-off. I have seen startups burn time building a full canary platform when a well-tuned rolling update would have been sufficient, and I have seen enterprises use rolling updates for a payment service where a canary would have caught a revenue-impacting bug earlier. The right choice depends on traffic volume, blast radius, cost tolerance, and how mature your observability is.
I like to frame the comparison around four questions: How much extra infrastructure do I need? How long are two versions running simultaneously? How quickly can I revert? And how much tooling do I need to operate it safely?
| Dimension | Rolling Update | Blue-Green | Canary |
|---|---|---|---|
| Extra Infrastructure Needed | Low — only surged pods (1-25% overhead) | High — full duplicate environment during switch | Low to medium — small extra capacity for canary subset |
| Time With Two Versions Live | Minutes, until rollout completes | Seconds to minutes during switch; otherwise isolated | Minutes to hours, intentionally prolonged for observation |
| Rollback Speed | Minutes — rolling back is another rolling update | Seconds — flip router back to previous environment | Seconds to minutes — shift weight back to 0% or abort |
| Traffic Granularity | None — pods replaced blindly, load balancer distributes | All or nothing — 0% or 100% on each environment | Fine-grained — 1% to 99% by weight, header, or geography |
| Operational Complexity | Low — built into Kubernetes/Terraform | Medium — needs two environments and router automation | High — needs mesh/ingress + metrics + automated analysis |
| Best For | Frequent, low-risk updates to stateless services | Major releases, strict validation, instant rollback needs | High-traffic services where gradual exposure reduces risk |
In my experience, the table above maps well to team maturity. If you are a team of five running a few services, rolling updates with solid health checks will carry you far. If you are handling financial transactions or cannot tolerate mixed-version behavior, invest in blue-green. If you serve thousands of requests per second and want data-driven confidence, invest in canary, but budget time for observability and traffic management. All three require immutable artifacts and reproducible infrastructure — the Docker Documentation guidance on tagging and building deterministic images is directly relevant here.
Matching Traffic Patterns and Release Risk to the Right Strategy
There is no single best strategy. I have run all three in the same organization for different services. Here is how I decide.
Use rolling updates when changes are backward compatible and frequent
If you deploy multiple times per day, have good test coverage, and your changes are mostly additive — new endpoints, bug fixes, dependency patches — rolling updates give you the best balance of safety and speed. Keep your replica count at 3 or higher, enforce readiness probes, and set maxUnavailable: 0. For stateful services like background workers consuming from a queue, ensure the worker can handle graceful shutdown; otherwise a pod termination will drop jobs.
Use blue-green when you need a clean cut and a fast undo button
Choose blue-green when the change is large and hard to test incrementally: framework upgrades, runtime version jumps, or migrations that require a full environment validation. It is also appropriate when your load is relatively low or you can afford the duplicate capacity for a short window. I have also recommended blue-green for teams that do not yet have good metrics for automated canary analysis — a human can manually verify the green environment by running smoke tests before promotion, which is simpler than building a statistical comparison pipeline.
Use canary when you need real-user validation at scale
Choose canary when the cost of a bad release is high and user behavior is hard to simulate. That includes user-facing APIs with diverse traffic, mobile backends where client versions vary, and ML model serving where live performance matters. Do not attempt canary with percentage-based replica counts alone if you need precise traffic control; use a mesh or ingress that supports weighted routing. Start with a conservative 5% step, automate promotion to 50% only after 10-15 minutes of clean metrics, and require manual approval for 100%. In my teams, we also run canaries only during business hours when engineers are available to respond.
A practical pattern I have seen succeed is to combine strategies. Use rolling updates as the baseline for all services, add blue-green for the two or three critical services where instant rollback is worth the cost, and build canary for the highest-traffic public APIs. Your CI/CD pipeline can support all three — the deployment stage simply calls a different template or Terraform module. If you are using Terraform to manage infrastructure, the Terraform Documentation shows how to parameterize environment Count and Service weights so the same code can drive any strategy.
Observability and Automated Rollback as the Safety Net for Every Strategy
No deployment strategy is safe without observability and an automatic way back. I have investigated incidents where the deployment mechanism worked perfectly, but nobody was watching the right metric, so a bad version ran for an hour before anyone noticed.
At a minimum, instrument four signals for every deployment: request error rate, p95/p99 latency, saturation (CPU/memory and queue depth), and business metrics (checkout success, login rate). Export these to Prometheus or your APM and create dashboards that compare the new version against the baseline. For canary and blue-green with preview services, label metrics by version so you can see version=v2 error rate isolated from version=v1.
Automating the Rollback Decision
Manual rollbacks rely on a human noticing and reacting, which fails when the on-call engineer is busy or the error is subtle. I recommend configuring automated rollbacks wherever your platform supports it. In Kubernetes, you can set progressDeadlineSeconds and let the Deployment fail if pods never become ready, but that only catches crash loops, not high error rates. Tools like Argo Rollouts and Flagger close the gap by querying your metrics provider.
A typical configuration I use is: canary step 10% for 5 minutes, analyze error rate < 1% and p95 < 400ms, then step to 50% for 10 minutes, then 100%. If any analysis fails, automatically roll back and mark the rollout as degraded. For blue-green, run a post-deployment job that executes synthetic transactions against the preview service; if any fail, do not promote. For rolling updates, I add a pipeline stage after the rollout that checks the same SLOs for five minutes and runs kubectl rollout undo if they breach.
Finally, log every deployment event — who triggered it, which image, which strategy, and the promotion/rollback decision — into your incident timeline. When you later debug a latency spike at 14:32, you will immediately see it correlated with a canary weight change. I have found that teams that treat deployments as observable events, not just pipeline steps, recover faster and learn more from each release.
Frequently Asked Questions
Which deployment strategy is safest for a beginner team with limited infrastructure?
Start with rolling updates. They are built into Kubernetes and most orchestrators, require no extra environments or service mesh, and keep costs low. Focus on getting readiness probes, health checks, and graceful shutdown right first. That alone eliminates most zero-downtime issues. Move to blue-green or canary only after you have reliable monitoring and need finer control over risk or faster rollback.
Do I need a service mesh to run canary deployments?
Not strictly, but you need some component that can split traffic by weight. For simple percentage-based canaries, you can run two Deployments with different replica counts behind a standard load balancer, though the distribution is approximate. For precise control, header-based routing, or automated analysis, a service mesh like Istio or Linkerd, or an ingress controller like NGINX Ingress or AWS Application Load Balancer with weighted target groups, is recommended. Many teams start without a mesh and add one when canary becomes a regular practice.
How do I handle database migrations with these strategies?
Make schema changes backward compatible and deploy them separately from code. The pattern is expand-then-contract: first deploy a migration that adds a new column or table without removing the old one, then deploy code that writes to both and reads from the new, then later remove the old column. This works for rolling updates and canary where two code versions run concurrently. Blue-green reduces the mixed-version window but still shares the database in most designs, so compatibility is still required. Never couple a breaking migration with an all-at-once code deploy.
Can I use feature flags instead of a deployment strategy?
They solve different problems and work best together. Deployment strategies control how a new artifact reaches infrastructure; feature flags control which users see new code paths inside that artifact. You can deploy with a rolling update and still use a flag to enable the feature for 5% of users — that gives you canary-like exposure without managing two versions. For major infrastructure changes or version incompatibilities, you still need a deployment strategy. In my experience, combining a simple deployment strategy with flags provides the most flexibility with the least operational overhead.