mongodb protocol_error ai_generated partial

MongoServerError: The resume token for namespace 'mydb.mycoll' is invalid or has been pruned: resumeAfter timestamp too old

ID: mongodb/change-stream-resume-token-invalid-for-namespace

Also available as: JSON · Markdown · 中文
70%Fix Rate
87%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
MongoDB 5.0 active — — —
MongoDB 6.0 active — — —
MongoDB 7.0 active — — —
Atlas M10+ active — — —

Root Cause

A change stream's resume token refers to an oplog entry that has been pruned due to the oplog size limit or because the change stream was idle longer than the oplog window (typically 24 hours on Atlas).

generic

中文

变更流的恢复令牌引用了已被修剪的操作日志条目，原因可能是操作日志大小限制，或变更流空闲时间超过操作日志窗口（Atlas 上通常为 24 小时）。

Official Documentation

https://www.mongodb.com/docs/manual/changeStreams/#resume-a-change-stream

Workarounds

  1. 70% success Start a new change stream from the current time and accept data loss of missed events. In code: `collection.watch([], { resumeAfter: null, startAfter: null })` or use `startAtOperationTime` with a recent timestamp. 
    Start a new change stream from the current time and accept data loss of missed events. In code: `collection.watch([], { resumeAfter: null, startAfter: null })` or use `startAtOperationTime` with a recent timestamp.
  2. 80% success If using Atlas, increase the oplog window by upgrading cluster tier or enabling long-term oplog retention (if available). For self-managed, increase oplog size: `db.adminCommand({ replSetResizeOplog: 1, size: 102400 })` (in MB). 
    If using Atlas, increase the oplog window by upgrading cluster tier or enabling long-term oplog retention (if available). For self-managed, increase oplog size: `db.adminCommand({ replSetResizeOplog: 1, size: 102400 })` (in MB).
  3. 75% success Implement a fallback mechanism: store the last processed document ID in an external store (e.g., Redis) and use `startAfter` with the last known _id to catch up, then switch to resume tokens. 
    Implement a fallback mechanism: store the last processed document ID in an external store (e.g., Redis) and use `startAfter` with the last known _id to catch up, then switch to resume tokens.

中文步骤

  1. 从当前时间开始新的变更流，接受丢失事件的数据损失。在代码中：`collection.watch([], { resumeAfter: null, startAfter: null })` 或使用带有最近时间戳的 `startAtOperationTime`。
  2. 如果使用 Atlas，通过升级集群层级或启用长期操作日志保留（如果可用）来增加操作日志窗口。对于自管理，增加操作日志大小：`db.adminCommand({ replSetResizeOplog: 1, size: 102400 })`（以 MB 为单位）。
  3. 实现回退机制：将最后处理的文档 ID 存储在外部存储（例如 Redis）中，并使用 `startAfter` 与最后已知的 _id 来追赶，然后切换回恢复令牌。

Dead Ends

Common approaches that don't work:

  1. 90% fail

    The resume token is already invalid; a new change stream must be started from the current time, losing all events between the old token and now.

  2. 100% fail

    The oplog entry is permanently pruned; retrying will always fail.

  3. 95% fail

    The error is server-side; no driver can recover a pruned oplog entry.