LOCK_ACQUIRE_TIMEOUT cicd runtime_error ai_generated true

Error: Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed (Service: DynamoDb, Status Code: 400)

ID: cicd/terraform-lock-acquire-timeout

Also available as: JSON · Markdown · 中文
85%Fix Rate
90%Confidence
1Evidence
2024-02-10First Seen

Root Cause

Terraform state lock acquisition fails due to concurrent operations on the same state file, typically when multiple CI jobs or team members run terraform apply simultaneously against the same backend (e.g., DynamoDB table).

generic

中文

由于对同一状态文件的并发操作，Terraform状态锁获取失败，通常发生在多个CI作业或团队成员同时针对同一后端（如DynamoDB表）运行terraform apply时。

Official Documentation

https://developer.hashicorp.com/terraform/language/state/locking

Workarounds

  1. 80% success Use terraform force-unlock <LOCK_ID> after verifying no other operation is running. First, identify lock ID: terraform force-unlock -force <LOCK_ID>. Example: terraform force-unlock -force 12345-abcde-67890-fghij 
    Use terraform force-unlock <LOCK_ID> after verifying no other operation is running. First, identify lock ID: terraform force-unlock -force <LOCK_ID>. Example: terraform force-unlock -force 12345-abcde-67890-fghij
  2. 95% success Implement a mutex in CI pipeline using a dedicated lock mechanism (e.g., GitLab CI resource_group or GitHub Actions concurrency) to prevent concurrent terraform apply runs. Example GitHub Actions: concurrency: group: terraform-deploy cancel-in-progress: false 
    Implement a mutex in CI pipeline using a dedicated lock mechanism (e.g., GitLab CI resource_group or GitHub Actions concurrency) to prevent concurrent terraform apply runs. Example GitHub Actions: concurrency: group: terraform-deploy cancel-in-progress: false

中文步骤

  1. Use terraform force-unlock <LOCK_ID> after verifying no other operation is running. First, identify lock ID: terraform force-unlock -force <LOCK_ID>. Example: terraform force-unlock -force 12345-abcde-67890-fghij
  2. Implement a mutex in CI pipeline using a dedicated lock mechanism (e.g., GitLab CI resource_group or GitHub Actions concurrency) to prevent concurrent terraform apply runs. Example GitHub Actions: concurrency: group: terraform-deploy cancel-in-progress: false

Dead Ends

Common approaches that don't work:

  1. Delete the lock file manually from S3 or DynamoDB to force unlock 60% fail

    Manual deletion can corrupt state if another operation is in progress, leading to data loss or inconsistent state. Terraform's force-unlock should be used instead.

  2. Set parallelism=1 in terraform config to serialize all operations 90% fail

    This only affects resource provisioning parallelism, not state lock contention from multiple processes; it does not prevent concurrent terraform apply commands.