huggingface type_error ai_generated true

类型错误：列 'labels' 期望特征类型 'Value(dtype='int32')'，但得到 'Value(dtype='int64')'。请将列转换为正确的数据类型。

TypeError: Expected feature type 'Value(dtype='int32')' but got 'Value(dtype='int64')' for column 'labels'. Cast the column to the correct dtype.

ID: huggingface/dataset-feature-type-mismatch

其他格式: JSON · Markdown 中文 · English
88%修复率
86%置信度
1证据数
2023-11-12首次发现

版本兼容性

版本状态引入弃用备注
datasets>=2.16.0 active — — —
transformers>=4.35.0 active — — —
Python>=3.8 active — — —

根因分析

数据集模式为某列定义了特定数据类型（如 int32），但实际加载的数据具有不同的数据类型（如 int64），导致批处理或模型输入准备时出现不匹配。

English

The dataset schema defines a specific dtype for a column (e.g., int32) but the actual data loaded has a different dtype (e.g., int64), causing a mismatch during batching or model input preparation.

generic

官方文档

https://huggingface.co/docs/datasets/en/features#features-types

解决方案

  1. Cast the column to the expected dtype using Dataset.cast_column:
from datasets import load_dataset, Features, Value
dataset = load_dataset('org/dataset', split='train')
expected_features = Features({'labels': Value('int32')})
dataset = dataset.cast(expected_features)
# Or cast a single column:
dataset = dataset.cast_column('labels', Value('int32'))
  2. Use Dataset.map to manually convert the column:
def convert_labels(example):
    example['labels'] = int(example['labels'])  # Python int is flexible
    return example
dataset = dataset.map(convert_labels)
# Then let the data collator handle casting automatically.

无效尝试

常见但无效的做法:

  1. 70% 失败

    Ignoring the error and using the dataset as-is may cause silent casting during training, leading to memory inefficiency or runtime errors in PyTorch.

  2. 80% 失败

    Dropping the column with remove_columns removes the feature entirely, causing a missing column error later.