huggingface data_error ai_generated true

ValueError: 数据集的特征与预期的模式不匹配。缺少列: ['text', 'label']。多余列: ['input', 'target']

ValueError: The features of the dataset do not match the expected schema. Missing columns: ['text', 'label']. Extra columns: ['input', 'target']

ID: huggingface/dataset-features-column-mismatch

其他格式: JSON · Markdown 中文 · English
90%修复率
85%置信度
1证据数
2023-08-10首次发现

版本兼容性

版本状态引入弃用备注
datasets>=2.10.0 active — — —
transformers>=4.25.0 active — — —

根因分析

从 Hugging Face Datasets 加载的数据集具有与训练脚本或分词器预期不同的列名。

English

Dataset loaded from Hugging Face Datasets has different column names than those expected by the training script or tokenizer.

generic

官方文档

https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.Dataset

解决方案

  1. Align columns using Dataset.rename_columns() and Dataset.remove_columns(): `dataset = dataset.rename_columns({'input': 'text', 'target': 'label'}).remove_columns(['unused_col'])`
  2. Use datasets.Dataset.map() with a function that selects only the required columns: `dataset = dataset.map(lambda x: {'text': x['input'], 'label': x['target']}, remove_columns=dataset.column_names)`
  3. Load the dataset with expected column names by specifying the 'columns' argument in load_dataset() if the dataset supports it, or create a new dataset with the correct schema.

无效尝试

常见但无效的做法:

  1. 40% 失败

    If there are more mismatches (e.g., 'target' vs 'label'), the error persists. Also, renaming may break other downstream code that expects 'input'.

  2. 50% 失败

    Trainer does not have ignore_columns; dropping columns with dataset.remove_columns() is correct but users often drop the wrong ones or forget to add missing columns.

  3. 70% 失败

    Model config does not control dataset schema; this is a data preprocessing issue, not a model architecture issue.