Remove Duplicate Entries from MongDB Collection

scottalanmiller

In MongoDB 2, there were easy ways to eliminate duplicates in collections. But now with MongoDB 3 and later, we need to do something a little more creative. This one is tested on MongoDB 3.4.

If you followed my other post about how to find the duplicates in the first place, we will be continuing on from that.

https://mangolassi.it/topic/16771/find-duplicate-value-entries-in-mongodb-collection

db.myCollection.aggregate([ 
    {$group: 
        { _id: "$field_to_search", "dups": { "$push": "$_id" }, count: { $sum: 1}}}, {$match: { count: { $gt: 1 }}}]).forEach(function(doc) { doc.dups.shift(); db.myCollection.remove({ "_id": {"$in": doc.dups }}); });

scottalanmiller

You can add an index as well, to avoid having duplicates occur in the future:

db.myCollection.createIndex({"my_field_to_be_unique":1},{unique:true})

bbigford

@scottalanmiller said in Remove Duplicate Entries from MongDB Collection:

You can add an index as well, to avoid having duplicates occur in the future:
db.myCollection.createIndex({"my_field_to_be_unique":1},{unique:true})

Good idea.