aws throttle_error ai_generated true

ProvisionedThroughputExceededException: Rate exceeded for shard

ID: aws/aws-kinesis-throughput-exceeded

Also available as: JSON · Markdown
85%Fix Rate
88%Confidence
80Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

Kinesis shard throughput limit exceeded. Each shard supports 1 MB/s write and 2 MB/s read.

generic

Workarounds

  1. 90% success Increase the number of shards or enable on-demand capacity mode 
    aws kinesis update-shard-count --stream-name NAME --target-shard-count N --scaling-type UNIFORM_SCALING or switch to ON_DEMAND mode

    Sources: https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-resharding.html

  2. 85% success Distribute records across shards using random or high-cardinality partition keys 
    Use UUID or composite keys as partition key to distribute records evenly across all shards

    Sources: https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

Dead Ends

Common approaches that don't work:

  1. Retry immediately without backoff 85% fail

    Aggressive retries without backoff amplify the problem and cause more throttling

  2. Use a single partition key for all records 90% fail

    Single partition key sends all records to one shard, creating a hot shard that will always throttle

Error Chain

Leads to:
Preceded by: