Copying a MongoDB database without leaving the cluster

Most of the times I have needed to duplicate a MongoDB database — seeding a staging environment from production, or snapshotting a database before a risky migration — I reached for a script that reads every document out of the source and writes it back into the destination. That works, but it round-trips every document through the application, which is slow and easy to get subtly wrong under load.

Let the cluster do the copying

MongoDB's aggregation pipeline has a $out stage that can write straight into a different database in the same cluster. Point an aggregation at a collection, end it with $out, and the server copies the result set itself — no document ever leaves the cluster to be re-serialised by a client.

collection.Aggregate(ctx, mongo.Pipeline{
  {{Key: "$out", Value: bson.D{
    {Key: "db", Value: dest},
    {Key: "coll", Value: name},
  }}},
})

mongo-db-syncer is a small Go CLI built around that one idea: list every collection in a source database, then run this pipeline against each one with a different destination database as the target.

Fan out, stop on the first failure

Databases can have dozens of collections, and there is no reason to copy them one at a time. Each collection gets its own goroutine, tracked by a sync.WaitGroup. The part I spent the most time getting right was what happens when one of them fails: I did not want twenty successful collections to finish quietly while one failed underneath them.

The fix was a single buffered error channel and a cancellable context shared by every worker. The first goroutine to fail sends its error and the context is cancelled; every other in-flight aggregation stops instead of running to a completion nobody asked for.

select {
case err := <-errChan:
  cancel()
  return err
case <-done:
  return nil
}

It is a small pattern, but I use a version of it often enough — fan out, wait for either everything finished or the first thing broke — that it was worth writing once, correctly, instead of copy-pasting a slightly wrong version of it into every side project.

Deliberately small

The whole tool is three flags: a Mongo URI, a source database and a destination database. No config file, no dry-run mode, no partial-collection filtering. go install gives you a binary, and the binary does the one thing it says on the readme.

The best tools I have built for myself are the ones I stopped adding features to.