# RuntimeError: NCCL error: version mismatch, expected 2.18.5 but got 2.19.1

- **ID:** `cuda/nccl-version-mismatch`
- **Domain:** cuda
- **Category:** config_error
- **Error Code:** `ncclSystemError`
- **Verification:** ai_generated
- **Fix Rate:** 90%

## Root Cause

The NCCL library version used at runtime differs from the one expected by PyTorch, often due to multiple NCCL installations or incorrect LD_LIBRARY_PATH.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| NCCL 2.18.5 | active | — | — |
| NCCL 2.19.1 | active | — | — |
| PyTorch 2.1.0 | active | — | — |

## Workarounds

1. **Set the environment variable `LD_LIBRARY_PATH` to point to the correct NCCL installation. For example: `export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nccl:$LD_LIBRARY_PATH`. Alternatively, use `conda install -c conda-forge nccl` to ensure consistency.** (90% success)
   ```
   Set the environment variable `LD_LIBRARY_PATH` to point to the correct NCCL installation. For example: `export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nccl:$LD_LIBRARY_PATH`. Alternatively, use `conda install -c conda-forge nccl` to ensure consistency.
   ```

## Dead Ends

- **** — Debugging does not resolve binary incompatibility. (95% fail)
- **** — PyTorch bundles its own NCCL, but system paths can override it. (70% fail)