ProvisionedThroughputExceededException terraform resource_error ai_generated true

Error: Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed: ProvisionedThroughputExceededException: Rate exceeded

ID: terraform/state-lock-dynamodb-throttling

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-04-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Terraform 1.5.0 active — — —
Terraform 1.6.0 active — — —
Terraform 1.7.0 active — — —

Root Cause

DynamoDB table used for state locking is experiencing throttling due to high request rates, exceeding its provisioned read/write capacity.

generic

中文

用于状态锁定的 DynamoDB 表因请求速率过高而受到限流，超出了预置的读写容量。

Official Documentation

https://developer.hashicorp.com/terraform/language/settings/backends/s3#dynamodb-table-configuration

Workarounds

  1. 85% success Increase the DynamoDB table's provisioned read and write capacity units (RCU/WCU) to handle the peak load. Switch to on-demand capacity if workload is unpredictable. 
    Increase the DynamoDB table's provisioned read and write capacity units (RCU/WCU) to handle the peak load. Switch to on-demand capacity if workload is unpredictable.
  2. 75% success Implement client-side retry logic by wrapping terraform commands in a bash script with exponential backoff. 
    Implement client-side retry logic by wrapping terraform commands in a bash script with exponential backoff.
  3. 70% success Reduce the frequency of terraform apply/plan operations during peak hours, or stagger multiple CI/CD pipelines to avoid concurrent lock attempts. 
    Reduce the frequency of terraform apply/plan operations during peak hours, or stagger multiple CI/CD pipelines to avoid concurrent lock attempts.

中文步骤

  1. 增加 DynamoDB 表的预置读写容量单元以处理峰值负载。如果工作负载不可预测，切换到按需容量模式。
  2. 通过将 terraform 命令包装在带有指数退避的 bash 脚本中，实现客户端重试逻辑。
  3. 在高峰时段减少 terraform apply/plan 操作的频率，或错开多个 CI/CD 管道以避免并发锁尝试。

Dead Ends

Common approaches that don't work:

  1. 90% fail

    The lock acquisition failed due to throttling, not because the lock is held. Force-unlock doesn't address the throughput issue.

  2. 80% fail

    Deleting the table removes all state lock items but also any existing locks. Recreating with default settings still has low throughput.

  3. 95% fail

    Terraform's retry mechanism is for provider operations, not for state lock acquisition. The lock mechanism is internal and not configurable.