huggingface runtime_error ai_generated true

ValueError: The number of outputs returned by the pipeline does not match the number of inputs. Expected 8 outputs, got 4.

ID: huggingface/pipeline-batch-inference-mismatch

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-08-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers>=4.30.0 active — — —
torch>=1.13.0 active — — —
python>=3.8 active — — —

Root Cause

When using pipeline with batch_size > 1 and a model that dynamically drops or merges samples (e.g., due to truncation or filtering in preprocess), the output count can diverge from input count.

generic

中文

使用 batch_size > 1 的管道且模型动态丢弃或合并样本（例如由于预处理中的截断或过滤）时，输出数量可能与输入数量不一致。

Official Documentation

https://huggingface.co/docs/transformers/main_classes/pipelines#batch-inference

Workarounds

  1. 70% success Disable any sample filtering in the pipeline's preprocess step (e.g., set `truncation=False` for summarization). Alternatively, use `return_full_text=False` in text-generation pipelines to avoid output count mismatch. 
    Disable any sample filtering in the pipeline's preprocess step (e.g., set `truncation=False` for summarization). Alternatively, use `return_full_text=False` in text-generation pipelines to avoid output count mismatch.
  2. 85% success Collect outputs in a list without batching and then manually batch inputs, ensuring each input produces exactly one output. Use `tokenizer.batch_decode` with `skip_special_tokens=True`. 
    Collect outputs in a list without batching and then manually batch inputs, ensuring each input produces exactly one output. Use `tokenizer.batch_decode` with `skip_special_tokens=True`.

中文步骤

  1. Disable any sample filtering in the pipeline's preprocess step (e.g., set `truncation=False` for summarization). Alternatively, use `return_full_text=False` in text-generation pipelines to avoid output count mismatch.
  2. Collect outputs in a list without batching and then manually batch inputs, ensuring each input produces exactly one output. Use `tokenizer.batch_decode` with `skip_special_tokens=True`.

Dead Ends

Common approaches that don't work:

  1. 40% fail

    Setting `batch_size=1` avoids the mismatch but kills performance for large datasets. The error is not caused by batch size per se but by sample filtering.

  2. 30% fail

    Manually overriding `__len__` in the dataset to match filtered outputs is fragile and breaks if filtering logic changes.