huggingface
type_error
ai_generated
true
TypeError：文本输入必须是 `str`（单个示例）、`List[str]`（批次）或 `List[List[str]]`（序列批次）类型。收到类型：<class 'int'>
TypeError: Text input must be of type `str` (single example), `List[str]` (batch) or `List[List[str]]` (batch of sequences). Received type: <class 'int'>
ID: huggingface/tokenizer-typeerror-nonstring
92%修复率
87%置信度
1证据数
2023-04-08首次发现
版本兼容性
| 版本 | 状态 | 引入 | 弃用 | 备注 |
|---|---|---|---|---|
| transformers>=4.20.0 | active | — | — | — |
根因分析
分词器接收到了非文本输入类型（例如整数、浮点数），而不是预期的字符串类型。
English
The tokenizer received a non-text input type (e.g., integer, float) instead of expected string types.
官方文档
https://huggingface.co/docs/transformers/v4.35.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__解决方案
-
在分词前将整数输入转换为字符串：对于单个整数使用 `tokenizer(str(123))`，对于整数列表使用 `tokenizer([str(x) for x in [1, 2, 3]])`。
-
确保数据管道输出字符串类型。如果使用数据集，通过 `dataset = dataset.map(lambda x: {'text': str(x['label'])})` 进行映射转换列。
无效尝试
常见但无效的做法:
-
70% 失败
While `str(123)` works for a single int, if the input is a list of ints like `[1, 2, 3]`, `str([1,2,3])` produces a single string '[1, 2, 3]' which the tokenizer will tokenize incorrectly as a single sequence of characters, not as three separate tokens.
-
95% 失败
`tokenizer.encode` also performs type checking and will raise the same error for non-string inputs. It is not a bypass.