huggingface type_error ai_generated true

TypeError: Text input must be of type `str` (single example), `List[str]` (batch) or `List[List[str]]` (batch of sequences). Received type: <class 'int'>

ID: huggingface/tokenizer-typeerror-nonstring

Also available as: JSON · Markdown · 中文

92%Fix Rate

87%Confidence

1Evidence

2023-04-08First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
transformers>=4.20.0	active	—	—	—

The tokenizer received a non-text input type (e.g., integer, float) instead of expected string types.

generic

分词器接收到了非文本输入类型（例如整数、浮点数），而不是预期的字符串类型。

90% success Convert integer inputs to strings before tokenization: `tokenizer(str(123))` for single ints, or `tokenizer([str(x) for x in [1, 2, 3]])` for lists of ints.
```
Convert integer inputs to strings before tokenization: `tokenizer(str(123))` for single ints, or `tokenizer([str(x) for x in [1, 2, 3]])` for lists of ints.
```
85% success Ensure your data pipeline outputs string types. If using a dataset, map with `dataset = dataset.map(lambda x: {'text': str(x['label'])})` to convert columns.
```
Ensure your data pipeline outputs string types. If using a dataset, map with `dataset = dataset.map(lambda x: {'text': str(x['label'])})` to convert columns.
```

在分词前将整数输入转换为字符串：对于单个整数使用 `tokenizer(str(123))`，对于整数列表使用 `tokenizer([str(x) for x in [1, 2, 3]])`。

确保数据管道输出字符串类型。如果使用数据集，通过 `dataset = dataset.map(lambda x: {'text': str(x['label'])})` 进行映射转换列。

Common approaches that don't work:

70% fail
While `str(123)` works for a single int, if the input is a list of ints like `[1, 2, 3]`, `str([1,2,3])` produces a single string '[1, 2, 3]' which the tokenizer will tokenize incorrectly as a single sequence of characters, not as three separate tokens.
95% fail
`tokenizer.encode` also performs type checking and will raise the same error for non-string inputs. It is not a bypass.