# RuntimeError: [torch.distributed] Broadcast failed on rank 3: ncclInternalError: Internal error - unexpected event

- **ID:** `pytorch/distributed-broadcast-error`
- **Domain:** pytorch
- **Category:** network_error
- **Verification:** ai_generated
- **Fix Rate:** 75%

## Root Cause

NCCL internal error during a broadcast collective operation, often caused by GPU hardware faults, driver issues, or network timeouts in multi-node setups.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| torch>=1.13 | active | — | — |
| torch<=2.5.1 | active | — | — |
| nccl>=2.14 | active | — | — |
| nccl<=2.22 | active | — | — |

## Workarounds

1. **Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py** (80% success)
   ```
   Set environment variable NCCL_IB_DISABLE=1 to disable InfiniBand and fall back to TCP, which is more stable on some clusters. For example: export NCCL_IB_DISABLE=1; torchrun --nproc_per_node=8 train.py
   ```
2. **Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118** (85% success)
   ```
   Update NCCL and CUDA drivers to compatible versions. Check NCCL compatibility matrix and use torch's bundled NCCL if possible: pip install torch --index-url https://download.pytorch.org/whl/cu118
   ```

## Dead Ends

- **Rebooting all nodes without changing NCCL configuration** — If the root cause is a driver mismatch or NCCL version incompatibility, a simple reboot will not fix the issue. (70% fail)
- **Increasing the number of GPUs per node to improve bandwidth** — More GPUs can exacerbate NCCL internal errors due to increased contention and resource pressure. (80% fail)