11000 mongodb data_error ai_generated true

MongoServerError: E11000 duplicate key error collection: test.users index: email_1 dup key: { email: null }

ID: mongodb/index-duplicate-key-on-sparse

Also available as: JSON · Markdown · 中文
85%Fix Rate
90%Confidence
1Evidence
2023-04-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
mongodb-3.6 active — — —
mongodb-4.0 active — — —
mongodb-4.2 active — — —
mongodb-4.4 active — — —
mongodb-5.0 active — — —
mongodb-6.0 active — — —
mongodb-7.0 active — — —

Root Cause

Multiple documents have null or missing values for a field with a unique sparse index, causing a duplicate key error because sparse indexes treat null and missing values as the same key.

generic

中文

多个文档在具有唯一稀疏索引的字段上具有空值或缺失值，导致重复键错误，因为稀疏索引将空值和缺失值视为相同的键。

Official Documentation

https://www.mongodb.com/docs/manual/core/index-sparse/

Workarounds

  1. 95% success Use a partial unique index instead of sparse to allow multiple nulls. Example: `db.users.createIndex({email:1},{unique:true,partialFilterExpression:{email:{$exists:true,$ne:null}}})` 
    Use a partial unique index instead of sparse to allow multiple nulls. Example: `db.users.createIndex({email:1},{unique:true,partialFilterExpression:{email:{$exists:true,$ne:null}}})`
  2. 85% success Update all documents with null email to a unique placeholder value, such as an empty string or a generated UUID. Example: `db.users.updateMany({email:null},{$set:{email:UUID().toString()}})` 
    Update all documents with null email to a unique placeholder value, such as an empty string or a generated UUID. Example: `db.users.updateMany({email:null},{$set:{email:UUID().toString()}})`
  3. 80% success Drop the sparse index and create a regular unique index, ensuring that null values are replaced with a sentinel value first. Example: `db.users.dropIndex('email_1'); db.users.createIndex({email:1},{unique:true})` 
    Drop the sparse index and create a regular unique index, ensuring that null values are replaced with a sentinel value first. Example: `db.users.dropIndex('email_1'); db.users.createIndex({email:1},{unique:true})`

中文步骤

  1. Use a partial unique index instead of sparse to allow multiple nulls. Example: `db.users.createIndex({email:1},{unique:true,partialFilterExpression:{email:{$exists:true,$ne:null}}})`
  2. Update all documents with null email to a unique placeholder value, such as an empty string or a generated UUID. Example: `db.users.updateMany({email:null},{$set:{email:UUID().toString()}})`
  3. Drop the sparse index and create a regular unique index, ensuring that null values are replaced with a sentinel value first. Example: `db.users.dropIndex('email_1'); db.users.createIndex({email:1},{unique:true})`

Dead Ends

Common approaches that don't work:

  1. 15% fail

    This removes data integrity guarantees for non-null values, potentially allowing unintended duplicates.

  2. 10% fail

    This is a temporary fix; future inserts with null will cause the same error.

  3. 20% fail

    Existing null documents remain, so the error persists until they are also updated.