huggingface data_error ai_generated true

KeyError: "Split 'validation' not found in dataset. Available splits: ['train', 'test']"

ID: huggingface/dataset-split-key-error

Also available as: JSON · Markdown · 中文
95%Fix Rate
90%Confidence
1Evidence
2023-04-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
datasets>=2.14.0 active — — —
transformers>=4.30.0 active — — —
Python>=3.8 active — — —

Root Cause

The requested dataset split name (e.g., 'validation') does not exist in the dataset configuration; the dataset may use a different naming convention like 'val' or 'test'.

generic

中文

请求的数据集分割名称（如 'validation'）在数据集配置中不存在；数据集可能使用不同的命名约定，如 'val' 或 'test'。

Official Documentation

https://huggingface.co/docs/datasets/en/loading#splits

Workarounds

  1. 95% success Check available splits before loading and use the correct name: from datasets import get_dataset_split_names splits = get_dataset_split_names('org/dataset') print(splits) # e.g., ['train', 'test'] # Then use the correct split: dataset = load_dataset('org/dataset', split='test') 
    Check available splits before loading and use the correct name:
from datasets import get_dataset_split_names
splits = get_dataset_split_names('org/dataset')
print(splits)  # e.g., ['train', 'test']
# Then use the correct split:
dataset = load_dataset('org/dataset', split='test')
  2. 90% success If no validation split exists, create one from the training data: from datasets import load_dataset dataset = load_dataset('org/dataset', split='train') train_val = dataset.train_test_split(test_size=0.1, seed=42) train_dataset = train_val['train'] val_dataset = train_val['test'] 
    If no validation split exists, create one from the training data:
from datasets import load_dataset
dataset = load_dataset('org/dataset', split='train')
train_val = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = train_val['train']
val_dataset = train_val['test']

中文步骤

  1. Check available splits before loading and use the correct name:
from datasets import get_dataset_split_names
splits = get_dataset_split_names('org/dataset')
print(splits)  # e.g., ['train', 'test']
# Then use the correct split:
dataset = load_dataset('org/dataset', split='test')
  2. If no validation split exists, create one from the training data:
from datasets import load_dataset
dataset = load_dataset('org/dataset', split='train')
train_val = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = train_val['train']
val_dataset = train_val['test']

Dead Ends

Common approaches that don't work:

  1. 60% fail

    Using split='all' to load all data and then manually splitting may cause data leakage if the dataset has predefined splits.

  2. 90% fail

    Renaming the split after loading with rename_column does not create a new split; it only renames a column.