cuda serialization_error ai_generated true

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False

ID: cuda/checkpoint-device-mismatch

Also available as: JSON · Markdown
96%Fix Rate
95%Confidence
80Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

A model checkpoint saved on a GPU machine is being loaded on a CPU-only machine (or a machine where CUDA is unavailable). torch.load() defaults to restoring tensors to their original device, which fails when that device does not exist.

generic

Workarounds

  1. 97% success Use map_location parameter in torch.load() to remap tensors to CPU 
    checkpoint = torch.load('model.pt', map_location=torch.device('cpu'))
model.load_state_dict(checkpoint)

    Sources: https://pytorch.org/docs/stable/generated/torch.load.html

  2. 95% success Use map_location='cpu' shorthand and then move to GPU if available 
    checkpoint = torch.load('model.pt', map_location='cpu')
model.load_state_dict(checkpoint)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)

    Sources: https://pytorch.org/tutorials/beginner/saving_loading_models.html

  3. 93% success Save checkpoints with device-agnostic state_dict instead of the full model 
    # When saving:
torch.save(model.state_dict(), 'model_weights.pt')
# When loading (works on any device):
model = MyModel()
model.load_state_dict(torch.load('model_weights.pt', map_location='cpu'))

    Sources: https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-state-dict-recommended

Dead Ends

Common approaches that don't work:

  1. Install CUDA toolkit on the CPU-only machine to make torch.cuda.is_available() return True 95% fail

    CUDA requires an NVIDIA GPU with driver support; installing the toolkit alone on a machine without a GPU will not make CUDA available

  2. Convert the checkpoint file manually by editing binary data 98% fail

    PyTorch checkpoint files use a complex pickle-based format; manual editing corrupts the file and loses tensor data

  3. Downgrade PyTorch version to bypass the device check 90% fail

    The device validation has existed across all modern PyTorch versions; downgrading introduces compatibility issues without fixing the root cause

Error Chain

Leads to:
Preceded by:
Frequently confused with: