RuntimeError: NCCL error: unhandled system error, NCCL version

Workarounds

1. 88% success Set NCCL_SOCKET_IFNAME to the correct network interface

   export NCCL_SOCKET_IFNAME=eth0

   Sources: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-socket-ifname

2. 85% success Update NVIDIA driver and NCCL to latest compatible versions

   # Check current versions:
   nvidia-smi  # Driver version
   nvcc --version  # CUDA version
   python -c "import torch; print(torch.cuda.nccl.version())"
   
   # Update driver:
   sudo apt-get install nvidia-driver-545
   # Update NCCL:
   conda install -c nvidia nccl

   Sources: https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html

3. 80% success Try NCCL_P2P_DISABLE=1 if peer-to-peer GPU communication fails

   export NCCL_P2P_DISABLE=1
   # Also try disabling IB if on InfiniBand:
   export NCCL_IB_DISABLE=1
   # Then rerun training:
   torchrun --nproc_per_node=4 train.py

   Sources: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html