huggingface type_error ai_generated true

TypeError: Expected feature type 'Value(dtype='int32')' but got 'Value(dtype='int64')' for column 'labels'. Cast the column to the correct dtype.

ID: huggingface/dataset-feature-type-mismatch

Also available as: JSON · Markdown · 中文

88%Fix Rate

86%Confidence

1Evidence

2023-11-12First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
datasets>=2.16.0	active	—	—	—
transformers>=4.35.0	active	—	—	—
Python>=3.8	active	—	—	—

Root Cause

The dataset schema defines a specific dtype for a column (e.g., int32) but the actual data loaded has a different dtype (e.g., int64), causing a mismatch during batching or model input preparation.

generic

中文

数据集模式为某列定义了特定数据类型（如 int32），但实际加载的数据具有不同的数据类型（如 int64），导致批处理或模型输入准备时出现不匹配。

Official Documentation

https://huggingface.co/docs/datasets/en/features#features-types

Workarounds

88% success Cast the column to the expected dtype using Dataset.cast_column: from datasets import load_dataset, Features, Value dataset = load_dataset('org/dataset', split='train') expected_features = Features({'labels': Value('int32')}) dataset = dataset.cast(expected_features) # Or cast a single column: dataset = dataset.cast_column('labels', Value('int32'))
```
Cast the column to the expected dtype using Dataset.cast_column:
from datasets import load_dataset, Features, Value
dataset = load_dataset('org/dataset', split='train')
expected_features = Features({'labels': Value('int32')})
dataset = dataset.cast(expected_features)
# Or cast a single column:
dataset = dataset.cast_column('labels', Value('int32'))
```
82% success Use Dataset.map to manually convert the column: def convert_labels(example): example['labels'] = int(example['labels']) # Python int is flexible return example dataset = dataset.map(convert_labels) # Then let the data collator handle casting automatically.
```
Use Dataset.map to manually convert the column:
def convert_labels(example):
    example['labels'] = int(example['labels'])  # Python int is flexible
    return example
dataset = dataset.map(convert_labels)
# Then let the data collator handle casting automatically.
```

中文步骤

Cast the column to the expected dtype using Dataset.cast_column:
from datasets import load_dataset, Features, Value
dataset = load_dataset('org/dataset', split='train')
expected_features = Features({'labels': Value('int32')})
dataset = dataset.cast(expected_features)
# Or cast a single column:
dataset = dataset.cast_column('labels', Value('int32'))

Use Dataset.map to manually convert the column:
def convert_labels(example):
    example['labels'] = int(example['labels'])  # Python int is flexible
    return example
dataset = dataset.map(convert_labels)
# Then let the data collator handle casting automatically.

Dead Ends

Common approaches that don't work:

70% fail
Ignoring the error and using the dataset as-is may cause silent casting during training, leading to memory inefficiency or runtime errors in PyTorch.
80% fail
Dropping the column with remove_columns removes the feature entirely, causing a missing column error later.