cloud pagination ai_generated true

S3 list_objects returns only 1000 keys even though bucket has more objects

ID: cloud/aws-s3-list-objects-1000-limit

Also available as: JSON · Markdown
92%Fix Rate
95%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

S3 ListObjectsV2 returns max 1000 keys per call by default. The 'IsTruncated' flag and 'NextContinuationToken' must be checked. Many tutorials omit the pagination loop, leading to silently incomplete results.

generic

Workarounds

  1. 95% success Use paginator to automatically handle continuation tokens 
    paginator = s3.get_paginator('list_objects_v2'); for page in paginator.paginate(Bucket='my-bucket'): ...
  2. 90% success Manually loop with ContinuationToken until IsTruncated is False 
    while True: resp = s3.list_objects_v2(Bucket=b, ContinuationToken=token); if not resp['IsTruncated']: break

Dead Ends

Common approaches that don't work:

  1. Call list_objects_v2 once and use all returned keys 95% fail

    Default MaxKeys is 1000. If bucket has more objects, you silently miss everything beyond the first 1000.

  2. Set MaxKeys to a very large number 88% fail

    MaxKeys cannot exceed 1000 per S3 API spec. Setting it higher is silently capped to 1000.