huggingface runtime_error ai_generated true

RuntimeError: Trainer callback hook 'on_log' failed with AttributeError: 'NoneType' object has no attribute 'step'

ID: huggingface/trainer-callback-hook-failure

Also available as: JSON · Markdown · 中文
80%Fix Rate
82%Confidence
1Evidence
2024-02-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
transformers>=4.35.0 active — — —

Root Cause

A custom callback's on_log method attempts to access trainer.state.global_step before the trainer state is fully initialized, typically when logging occurs during the first step.

generic

中文

自定义回调的on_log方法在训练器状态完全初始化之前尝试访问trainer.state.global_step，通常发生在第一步记录日志时。

Official Documentation

https://huggingface.co/docs/transformers/main/en/main_classes/callback

Workarounds

  1. 95% success Add a guard in the callback to check if trainer.state is initialized: `if trainer.state.global_step is not None:` before accessing step. 
    Add a guard in the callback to check if trainer.state is initialized: `if trainer.state.global_step is not None:` before accessing step.
  2. 85% success Override the on_step_begin callback instead of on_log, as state is guaranteed to be initialized at step start. 
    Override the on_step_begin callback instead of on_log, as state is guaranteed to be initialized at step start.
  3. 90% success Initialize the callback with a default step value: `self.step = 0` in __init__, then use self.step instead of trainer.state.global_step. 
    Initialize the callback with a default step value: `self.step = 0` in __init__, then use self.step instead of trainer.state.global_step.

中文步骤

  1. Add a guard in the callback to check if trainer.state is initialized: `if trainer.state.global_step is not None:` before accessing step.
  2. Override the on_step_begin callback instead of on_log, as state is guaranteed to be initialized at step start.
  3. Initialize the callback with a default step value: `self.step = 0` in __init__, then use self.step instead of trainer.state.global_step.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    The root cause is accessing state before initialization; moving to another callback may still encounter the same issue if state is not ready.

  2. 90% fail

    This only postpones the error; it will still occur at the first log event regardless of step count.

  3. 85% fail

    The error is raised by the Trainer's callback execution wrapper; catching it inside the callback does not prevent the Trainer from propagating it.