Terraform state is the single source of truth that maps your configuration to the real infrastructure it manages, and how you store that state determines whether your team can collaborate safely or spends its time untangling drift and corruption. In my early projects I kept terraform.tfstate on a laptop and passed it around via encrypted zip files, which worked until two engineers applied at the same time and overwrote each other's changes to a production VPC. After that incident I moved every non-trivial environment to a remote backend and never looked back. This article covers the practical mechanics of remote backends, how state locking actually works under the hood, and the exact process I use to migrate state between backends without downtime, based on running Terraform for teams of 5 to 40 engineers on AWS, GCP, and Azure. If you are structuring your codebase alongside your state strategy, it helps to pair this with Terraform Modules: Design Patterns, Versioning and Registry Publishing so your state boundaries align with module boundaries.
Why Local State Breaks Down Beyond a Single Workstation
By default Terraform writes a file called terraform.tfstate in your working directory in JSON format. That file contains a complete snapshot of every resource attribute, including sensitive values, dependencies, and the lineage metadata Terraform uses to plan diffs. For a solo developer prototyping on a single machine, local state is simple and fast. The problems start as soon as you need history, sharing, or safety.
Local state has no concurrency control. If two people run terraform apply against the same local file stored in Git or on a shared drive, the last write wins and the other person's changes are silently lost. I have seen this cause duplicate EC2 instances, orphaned RDS databases that were removed from state but not destroyed, and security groups that drifted for weeks because the local state no longer reflected reality. Local state also forces you to solve secrets management yourself. That JSON file will contain database passwords, private keys, and access keys in plaintext unless you have marked variables as sensitive, and even then they are still present in the file.
The hidden risks of committing state to version control
Some teams try to commit terraform.tfstate to Git to get versioning. This creates a worse set of problems: merge conflicts on a generated JSON file are unresolvable by hand, every clone of the repo now contains secrets, and git history retains those secrets forever even if you later delete the file. Remote state solves this by moving the state file out of the developer workstation entirely and putting it behind an API that provides locking, encryption, and access control. The Terraform Documentation is explicit that remote backends are the recommended path for any collaborative environment, and in my experience that recommendation should be treated as a requirement once more than one person can apply.
What remote state actually stores and why it matters
Whether local or remote, the state file contains more than just resource IDs. It stores the full resource graph, outputs, data source results, and Terraform version metadata. This is what allows Terraform to determine that changing an instance_type requires replacement while changing tags only requires an in-place update. When you move to a remote backend, you are not just moving a file; you are moving the coordination point for all plan and apply operations. Understanding that helps explain why locking and migration need careful handling.
Configuring the S3 Backend with DynamoDB Locking in Practice
On AWS, the most common remote backend pattern is an S3 bucket for storage plus a DynamoDB table for locking. I have deployed this pattern for more than fifty environments and it remains the most robust self-managed option if you are not using Terraform Cloud. The S3 bucket provides versioning, encryption, and a durable store for the state JSON, while DynamoDB provides a strongly consistent lock via conditional writes.
Start by creating the bucket and table outside of the Terraform configuration that will use them. You do not want the infrastructure that stores your state to be managed by the same state it stores, or you will create a chicken-and-egg problem on first creation and on destruction. I typically create these with a bootstrap Terraform configuration or via CloudFormation that is used only once per account.
# bootstrap/backend-bootstrap.tf - applied once per account/region
resource "aws_s3_bucket" "tf_state" {
bucket = "acme-terraform-state-prod-us-east-1"
}
resource "aws_s3_bucket_versioning" "tf_state" {
bucket = aws_s3_bucket.tf_state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state" {
bucket = aws_s3_bucket.tf_state.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.tf_state.arn
}
}
}
resource "aws_dynamodb_table" "tf_lock" {
name = "acme-terraform-locks"
billing_mode = "PAY_PER_REQUEST"
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
With the bucket and table in place, configure the backend in your main configuration. I prefer to keep the backend block minimal and pass environment-specific values via a backend configuration file at init time. This keeps the same code portable across accounts and regions without hardcoding bucket names in version control.
# main.tf
terraform {
backend "s3" {
# Intentionally left empty - values supplied via -backend-config
}
required_version = ">= 1.5.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
# backend.hcl (per environment, not committed with secrets if using role)
bucket = "acme-terraform-state-prod-us-east-1"
key = "networking/prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "acme-terraform-locks"
encrypt = true
kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abcd-1234"
workspace_key_prefix = "workspaces"
Initialize with terraform init -backend-config=backend.hcl. Once initialized, Terraform will no longer create a local terraform.tfstate file; instead it will read and write directly to S3. You can verify the setup by running terraform plan, then checking the S3 object and its versions. In my experience, you should also enable S3 bucket replication and MFA delete for production state buckets, and restrict access with a bucket policy that denies uploads without KMS encryption and requires TLS. DynamoDB locking is automatic: when terraform plan or apply starts, Terraform writes a lock item with a unique ID, your user info, and a timestamp. If another process tries to acquire the lock while it is held, it will fail fast with a clear error rather than corrupting state.
Comparing Remote Backend Architectures: S3, GCS, AzureRM and Terraform Cloud
While S3 with DynamoDB is the default on AWS, GCP and Azure have equivalent native backends that integrate with their own storage and locking primitives. Terraform Cloud and Terraform Enterprise provide a fully managed alternative where HashiCorp handles the storage, locking, run history, and access controls for you. Choosing between them is less about features and more about where your team already operates and what operational burden you want to carry.
I have used GCS and AzureRM backends on multi-cloud projects and they behave similarly from Terraform's perspective. The GCS backend uses a single bucket with object versioning and relies on GCS's native object locking, so you do not need a separate lock table. The azurerm backend uses an Azure Storage Account blob container and a blob lease for locking. Terraform Cloud goes further by adding remote execution, so terraform plan and apply run on HashiCorp's runners with variables injected from its encrypted store, which reduces the need to distribute provider credentials to developers. For teams already invested in GitOps workflows, that remote execution model pairs well with approaches described in GitOps with ArgoCD and Flux: Declarative Kubernetes Deployment, where state changes are triggered by pipeline rather than local CLI.
| Backend | Storage Layer | Locking Mechanism | Native Versioning & Encryption | When I Recommend It |
|---|---|---|---|---|
| s3 | S3 bucket, key per state file | DynamoDB table with conditional writes on LockID | S3 Versioning + SSE-S3/SSE-KMS | AWS-native teams needing full control, cross-account state isolation |
| gcs | GCS bucket, prefix per workspace | GCS object metadata lock, no extra table | GCS Versioning + Google-managed or CMEK | GCP-native teams, simplest self-managed backend to operate |
| azurerm | Azure Blob Storage container | Blob lease (60s with renewal) | Blob versioning + Storage Service Encryption | Azure-native teams with strict private networking via storage firewall |
| remote (Terraform Cloud) | Managed by HashiCorp, per-workspace | Managed run queue, per-workspace locking | Managed, encrypted at rest, full run history | Teams wanting SSO, policy sets, remote runs and no bucket management |
From an operational perspective, the self-managed options require you to own backup, lifecycle, and access auditing. I set S3 and GCS bucket lifecycle rules to retain at least 90 days of versions and replicate production state buckets to a second region. For Terraform Cloud, those concerns shift to workspace permissions and API token scoping. One practical difference I have noticed: Terraform Cloud's remote backend enforces a single active run per workspace by default, which prevents concurrent applies more strictly than S3, where you could technically have two different state keys in the same bucket racing if your key naming is inconsistent. Consistent key naming like env/component/terraform.tfstate avoids that class entirely.
Diagnosing and Resolving State Locking Contention and Force-Unlock Scenarios
Locking is what keeps state safe, but it is also the most common source of confusion for engineers new to remote backends. A typical error looks like: Error: Error acquiring the state lock: ConditionalCheckFailedException. This means another Terraform process currently holds the lock. In normal use this is expected behavior - you should wait for the other run to finish. The problem cases are when a process crashes, a CI runner is terminated, or a network partition leaves a stale lock behind.
Terraform stores lock information as an item in your DynamoDB table, a metadata object in GCS, or a lease on the blob in Azure. You can inspect the lock directly before taking any destructive action. For S3, I check the DynamoDB item using the AWS CLI and compare the timestamp and the Operation field to what my team was doing. If the lock is more than 15 minutes old and no CI job is actively running, it is likely stale.
# Inspecting and safely removing a stale S3 DynamoDB lock
aws dynamodb get-item \
--table-name acme-terraform-locks \
--key '{"LockID": {"S": "acme-terraform-state-prod-us-east-1/networking/prod/terraform.tfstate-md5"}}'
# Only run this if you have confirmed no active apply is running
terraform force-unlock -force 7a3f8b1c-2d4e-4a9b-9c0e-1234567890ab
# Alternative with explicit backend config if working_directory differs
terraform force-unlock -backend-config=backend.hcl 7a3f8b1c-2d4e-4a9b-9c0e-1234567890ab
I have found that force-unlock should be treated as a break-glass operation, not a routine fix. You must be certain that the original process is truly dead, otherwise you can corrupt state when two writers push different serial numbers. In CI, I mitigate stale locks by setting a timeout on the pipeline job and ensuring the runner always runs terraform apply with -lock-timeout=10m so it waits rather than failing immediately under contention. Another pattern I use is to wrap CI applies with a trap that always attempts to unlock on exit, and to run Terraform with detailed logs so the lock ID is captured in the pipeline output for later diagnosis. For long-term prevention, split large monolithic states into smaller, component-scoped states; smaller blast radius means shorter lock durations and fewer collisions. This naturally pushes you toward a module structure covered in Pulumi vs Terraform: Infrastructure as Real Code vs Configuration discussions about composition, even when staying within Terraform itself.
Executing Zero-Downtime State Migration Between Backends
State migration becomes necessary when you rename a state key, consolidate multiple states, move from S3 to Terraform Cloud, or split a monolith. Terraform provides a controlled migration path via terraform init -migrate-state and -reconfigure flags, but the behavior is subtle and worth practicing in a non-production environment first.
The safest workflow I use involves four phases: backup, reconfigure, migrate, and verify. First, pull a local backup of the current state with terraform state pull > backup-$(date +%F).json and also verify that S3 versioning shows a recent version. Second, update your backend configuration to point at the new location but do not run init yet. Third, run terraform init -migrate-state. Terraform will detect that the backend configuration has changed, show a diff of the old versus new backend, and ask for confirmation to copy state. It copies the state file to the new backend, verifies the checksum, and only then updates your local .terraform/terraform.tfstate cache. It does not delete the old state object, which gives you a rollback path.
# Migrating from S3 key v1 to S3 key v2 (or to another backend)
# 1. Backup current state
terraform state pull > state-backup-2024-12-19.json
aws s3api get-object --bucket acme-terraform-state-prod-us-east-1 \
--key networking/prod/terraform.tfstate state-s3-backup.json
# 2. Update backend.hcl with new key
# bucket = "acme-terraform-state-prod-us-east-1"
# key = "networking/prod-v2/terraform.tfstate"
# 3. Migrate
terraform init -migrate-state -backend-config=backend.hcl
# Respond 'yes' when prompted: "Do you want to copy existing state to the new backend?"
# 4. Verify and use moved state operations if refactoring resources
terraform plan
terraform state list
terraform state mv 'aws_instance.web' 'aws_instance.web_new'
If you are refactoring resources at the same time, for example moving an aws_instance from a root module into a child module, use terraform state mv rather than tainting or importing. The state mv command rewrites the state JSON without touching the real infrastructure, so the next plan will show as no changes if done correctly. I've found that state mv across workspaces or backends requires you to pull state locally, edit, and push, which is error prone; instead keep mv operations within the same backend and migrate the entire state before splitting.
Handling workspaces and partial migration pitfalls
When using workspaces, remember that each workspace is stored as a separate key or prefix under workspace_key_prefix. Migrating without specifying the correct workspace will migrate only the default workspace and leave others behind. Always run terraform workspace list and terraform workspace select before init, and verify each workspace after migration with terraform workspace select staging && terraform state list. In my experience, the most common migration failure is partial permissions on the new bucket: Terraform can write the new state but fails to acquire the lock table on the first apply, leaving two sources of truth. Test the new backend with a no-op plan immediately after migration and confirm lock table writes before announcing completion. For a deeper validation workflow after migration, integrate automated policy checks as described in Infrastructure Testing: Terratest, Checkov and Policy-as-Code Validation to ensure no security drift was introduced during the move.
Securing State Files: Encryption, Access Controls and Secret Leakage Prevention
State files contain every sensitive value Terraform manages, including rds instance passwords, tls private keys, and provider tokens if they were ever stored as variables. Treating state as a secret by default is the only sustainable posture. On S3, that means SSE-KMS with a dedicated KMS key per environment, bucket policies that enforce aws:SecureTransport and s3:x-amz-server-side-encryption, and versioning to recover from accidental overwrite. I also enable S3 access logging to a separate audit bucket and enforce that only a narrow IAM role used by CI can read and write state, while developers assume that role via SSO with short-lived credentials.
For Terraform Cloud, the equivalent is workspace execution mode set to remote, variables marked as sensitive, and team permissions set to per-workspace rather than organization-wide. In either model, avoid passing secrets through terraform.tfvars in plaintext. Use Vault, AWS Secrets Manager, or Terraform Cloud's sensitive variables injected at run time. I have audited teams where secrets lingered in state for years because they were originally passed as a plain string; to remediate, rotate the secret after you move to a secure source, then perform a targeted terraform apply so the new value overwrites the old one in state, and finally consider migrating to a new state key so old S3 versions age out via lifecycle policy.
Access control should mirror your blast radius. A platform team should not have write access to a production network state if they only manage development clusters. I map one IAM role or Terraform Cloud team per state file, use key prefixes to isolate by component, and log every terraform state pull via CloudTrail data events or Terraform Cloud audit logs. Encryption alone does not prevent leakage if any engineer can pull state to their laptop and upload it elsewhere, so combine least privilege with detection.
Frequently Asked Questions
Can I change backend configuration without migrating state?
Yes, with terraform init -reconfigure. This flag tells Terraform to re-initialize the backend using the new configuration without copying state. Use it when you are only changing backend settings like region or dynamodb_table reference but the state key itself remains the same, or when you have already manually copied the state object. If you use -reconfigure when the key has changed, Terraform will initialize an empty state at the new location and your next plan will propose to recreate everything, which is destructive. When in doubt, use -migrate-state.
What happens if I lose the DynamoDB lock table for an S3 backend?
Terraform can still read and write the S3 state file without the lock table, but it will do so without concurrency protection. Any concurrent apply will risk corruption. If the table is accidentally deleted, recreate it with the same name and hash key LockID. Existing state files in S3 remain valid and the next plan will create a new lock item. I recommend importing the lock table itself into a bootstrap state and enabling point-in-time recovery so you can restore it without re-creating.
How do I move resources between states without destroying them?
Use terraform state mv in the source state to remove the binding and terraform import or a second state mv in the destination if you have pulled state locally. The safer approach for bulk moves is to use terraform state pull, manually edit the JSON to move the resource blocks between files, then terraform state push after verifying the serial numbers. I only do manual JSON edits when moving dozens of resources; for one or two resources, state mv within the same backend after you have split the configuration is cleaner and less error prone.
Should I enable S3 versioning and replication for Terraform state?
Yes for versioning, and yes for replication if the state is production-critical. Versioning gives you immediate rollback for accidental state push or corruption; I keep versioning enabled and use a lifecycle rule to expire noncurrent versions after 90 days. Replication to a second region protects against regional S3 issues and provides a recovery path independent of Terraform. Combine both with SSE-KMS and blocking public access to ensure durability does not come at the cost of exposure.