cuda distributed_computing ai_generated true

RuntimeError: NCCL error: invalid argument - ncclInvalidArgument

ID: cuda/nccl-init-rank-mismatch

Also available as: JSON · Markdown
82%Fix Rate
85%Confidence
70Evidence
2022-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

NCCL initialization fails because the world_size, rank, or MASTER_ADDR/MASTER_PORT environment variables are misconfigured. The ranks disagree about the total number of participants, or a rank ID is out of range.

generic

Workarounds

  1. 90% success Ensure WORLD_SIZE equals the actual number of processes and each RANK is unique in [0, WORLD_SIZE) 
    # For torchrun (recommended):
torchrun --nproc_per_node=4 --nnodes=1 train.py
# torchrun sets RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, MASTER_PORT automatically

# For manual launch, verify:
# WORLD_SIZE = total GPUs across all nodes
# RANK = unique per-process (0 to WORLD_SIZE-1)
# LOCAL_RANK = GPU index on this node

    Sources: https://pytorch.org/docs/stable/elastic/run.html

  2. 88% success Use torchrun instead of manual dist.init_process_group() with environment variables 
    # Replace:
# python -m torch.distributed.launch --nproc_per_node=4 train.py
# With:
torchrun --nproc_per_node=4 --nnodes=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:29500 train.py

# In the script:
import torch.distributed as dist
dist.init_process_group(backend='nccl')  # torchrun handles env vars

    Sources: https://pytorch.org/docs/stable/elastic/run.html

  3. 85% success For multi-node, verify MASTER_ADDR is reachable from all nodes and MASTER_PORT is open 
    # On each worker node, verify connectivity:
nc -zv $MASTER_ADDR $MASTER_PORT
# MASTER_ADDR should be the IP of rank 0 node, not 'localhost'
# Ensure firewall allows the port range (default 29500)

Dead Ends

Common approaches that don't work:

  1. Setting NCCL_DEBUG=INFO without checking environment variables first 60% fail

    While NCCL_DEBUG provides useful logging, the root cause is almost always that WORLD_SIZE, RANK, MASTER_ADDR, or MASTER_PORT are set incorrectly. Debug logs will just confirm the mismatch without explaining why the environment is wrong.

  2. Upgrading NCCL version to fix the 'invalid argument' error 80% fail

    ncclInvalidArgument is a parameter validation error, not a bug. The communicator creation rejects the configuration because rank >= world_size or world_size doesn't match across processes.

Error Chain

Leads to:
Frequently confused with: