huggingface runtime_error ai_generated true

RuntimeError：期望所有张量在相同设备上，但模型参数已被卸载到CPU

RuntimeError: expected all tensors to be on the same device, but model parameters have been offloaded to CPU

ID: huggingface/model-offloaded-to-cpu

其他格式: JSON · Markdown 中文 · English
80%修复率
85%置信度
1证据数
2024-03-15首次发现

版本兼容性

版本状态引入弃用备注
transformers>=4.36.0 active — — —
accelerate>=0.25.0 active — — —
PyTorch>=2.1.0 active — — —

根因分析

模型使用device_map='auto'或类似卸载策略加载，但前向传播等操作在没有正确设备放置的情况下在另一个设备上执行。

English

Model was loaded with device_map='auto' or similar offloading, but an operation (e.g., forward pass) is being performed on a different device without proper device placement.

generic

官方文档

https://huggingface.co/docs/accelerate/v0.28.0/en/usage_guides/big_modeling

解决方案

  1. Ensure all operations use the same device context. Use accelerate's context manager: from accelerate import dispatch_model; dispatch_model(model, device_map='auto') and then run inference within a with torch.no_grad() block on the same device.
  2. Set device_map to a specific device (e.g., device_map='cuda:0') instead of 'auto' to force all layers onto GPU if memory permits.

无效尝试

常见但无效的做法:

  1. Manually moving model to GPU with model.to('cuda') after loading with device_map 70% 失败

    device_map='auto' already handles placement; model.to() overrides it and may cause offloaded layers to be lost or cause memory issues on large models.

  2. Setting device_map=None and manually placing model on GPU 85% 失败

    For very large models that require offloading, disabling device_map may cause OOM because the entire model won't fit on GPU.