huggingface type_error ai_generated true

ValueError: Text input must of type str (single example), List[str] (batch)

ID: huggingface/wrong-input-type

Also available as: JSON · Markdown
92%Fix Rate
92%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Tokenizer received wrong input type. Usually passing raw data instead of text strings.

generic

Workarounds

  1. 95% success Pass correct type: str for single, List[str] for batch 
    tokenizer('single text') or tokenizer(['text1', 'text2'])

    Sources: https://huggingface.co/docs/transformers/

  2. 90% success Use tokenizer with padding and truncation for batches 
    tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
  3. 85% success Check the input type before calling tokenizer

Dead Ends

Common approaches that don't work:

  1. Convert everything to string with str() 82% fail

    str() on a list produces '['text1', 'text2']' which is tokenized as one string, not a batch

  2. Wrap single string in a list unconditionally 55% fail

    Some pipeline functions handle single strings differently from batches; check the API first