python system_error ai_generated partial

MemoryError: Unable to allocate array with shape (100000000,) and data type float64

ID: python/starlette-memory-error-upload

Also available as: JSON · Markdown · 中文
80%Fix Rate
82%Confidence
0Evidence
2024-05-11First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
3.x active — — —

Root Cause

Attempting to read a large file upload into memory all at once in a Starlette/FastAPI endpoint.

generic

中文

在 Starlette/FastAPI 端点中尝试一次性将大文件上传读入内存。

Workarounds

  1. 85% success 
    Stream the file using UploadFile.read(chunk_size) in a loop and process chunks: while chunk := await file.read(1024*1024): process(chunk)
  2. 80% success 
    Use a temporary file on disk: with tempfile.NamedTemporaryFile(delete=False) as tmp: shutil.copyfileobj(file.file, tmp)

Dead Ends

Common approaches that don't work:

  1. 90% fail

    These don't affect Python's heap allocation for large numpy arrays; physical RAM is exhausted.

  2. 80% fail

    This loads the whole file into a bytes object, still causing memory pressure.