huggingface config_error ai_generated true

RuntimeError: You cannot set both `device_map` and `device` when using the `accelerate` launcher with multiple GPUs.

ID: huggingface/accelerate-multi-gpu-device-map-conflict

Also available as: JSON · Markdown · 中文
90%Fix Rate
86%Confidence
1Evidence
2023-05-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
accelerate>=0.20.0 active — — —
transformers>=4.30.0 active — — —

Root Cause

When using accelerate launcher with multiple GPUs, the device is automatically managed; providing both device_map and device arguments causes a conflict in device placement.

generic

中文

在使用 accelerate 启动器且多 GPU 时，设备是自动管理的；同时提供 device_map 和 device 参数会导致设备放置冲突。

Official Documentation

https://huggingface.co/docs/accelerate/en/package_reference/launcher

Workarounds

  1. 95% success Remove the `device` argument and only use `device_map='auto'` (or a custom dict) when loading the model. Example: `model = AutoModelForCausalLM.from_pretrained('model-name', device_map='auto')`. The accelerate launcher will handle multi-GPU placement automatically. 
    Remove the `device` argument and only use `device_map='auto'` (or a custom dict) when loading the model. Example: `model = AutoModelForCausalLM.from_pretrained('model-name', device_map='auto')`. The accelerate launcher will handle multi-GPU placement automatically.
  2. 80% success If you must set a specific device, do not use the accelerate launcher; instead, use `with torch.device('cuda:0'): model = ...` and manually wrap with DataParallel or DistributedDataParallel. 
    If you must set a specific device, do not use the accelerate launcher; instead, use `with torch.device('cuda:0'): model = ...` and manually wrap with DataParallel or DistributedDataParallel.
  3. 90% success Use `accelerate launch` without any device_map or device argument in the script; let accelerate handle device placement via its config file (e.g., `--num_processes=4`). 
    Use `accelerate launch` without any device_map or device argument in the script; let accelerate handle device placement via its config file (e.g., `--num_processes=4`).

中文步骤

  1. Remove the `device` argument and only use `device_map='auto'` (or a custom dict) when loading the model. Example: `model = AutoModelForCausalLM.from_pretrained('model-name', device_map='auto')`. The accelerate launcher will handle multi-GPU placement automatically.
  2. If you must set a specific device, do not use the accelerate launcher; instead, use `with torch.device('cuda:0'): model = ...` and manually wrap with DataParallel or DistributedDataParallel.
  3. Use `accelerate launch` without any device_map or device argument in the script; let accelerate handle device placement via its config file (e.g., `--num_processes=4`).

Dead Ends

Common approaches that don't work:

  1. 80% fail

    The error is raised explicitly; both arguments are passed and cause a conflict in the model loading logic.

  2. 50% fail

    With device=0, the model is placed only on GPU 0, wasting other GPUs and potentially causing OOM on GPU 0.

  3. 40% fail

    This bypasses accelerate's device management entirely, causing the model to be on a single GPU and not utilizing multi-GPU parallelism.