1114 aws resource_error ai_generated true

MySQL server has gone away: error 1114 (HY000) The table is full - Storage autoscaling failed to increase capacity

ID: aws/rds-storage-full-autoscaling-failure

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2024-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
MySQL 8.0.35 active — — —
RDS 2024-09-01 active — — —
AWS CLI 2.18.0 active — — —

Root Cause

RDS instance storage is full and automatic storage scaling failed due to reaching the maximum storage limit, insufficient IAM permissions for the autoscaling policy, or a storage type that does not support autoscaling (e.g., magnetic).

generic

中文

RDS 实例存储已满，由于达到最大存储限制、自动扩展策略的 IAM 权限不足或存储类型不支持自动扩展（例如，磁性存储），自动存储扩展失败。

Official Documentation

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageAutoscaling.html

Workarounds

  1. 95% success Increase storage manually via the AWS CLI: `aws rds modify-db-instance --db-instance-identifier mydb --allocated-storage 200 --apply-immediately` 
    Increase storage manually via the AWS CLI: `aws rds modify-db-instance --db-instance-identifier mydb --allocated-storage 200 --apply-immediately`
  2. 85% success Enable storage autoscaling with a higher maximum limit: `aws rds modify-db-instance --db-instance-identifier mydb --max-allocated-storage 1000` 
    Enable storage autoscaling with a higher maximum limit: `aws rds modify-db-instance --db-instance-identifier mydb --max-allocated-storage 1000`
  3. 70% success Run OPTIMIZE TABLE on the affected table to reclaim space: `OPTIMIZE TABLE mytable;` (note: this locks the table and may take time). 
    Run OPTIMIZE TABLE on the affected table to reclaim space: `OPTIMIZE TABLE mytable;` (note: this locks the table and may take time).

中文步骤

  1. Increase storage manually via the AWS CLI: `aws rds modify-db-instance --db-instance-identifier mydb --allocated-storage 200 --apply-immediately`
  2. Enable storage autoscaling with a higher maximum limit: `aws rds modify-db-instance --db-instance-identifier mydb --max-allocated-storage 1000`
  3. Run OPTIMIZE TABLE on the affected table to reclaim space: `OPTIMIZE TABLE mytable;` (note: this locks the table and may take time).

Dead Ends

Common approaches that don't work:

  1. 75% fail

    InnoDB does not release disk space to the OS after deletes; the table remains full until OPTIMIZE is run.

  2. 90% fail

    Storage is still full; the error will reappear immediately after restart.

  3. 60% fail

    If storage fills again, autoscaling will fail if not re-enabled or if the new limit is still too low.