r/mongodb • u/mr_pants99 • 5h ago
mongoexport is great for one collection. What do you use for multi-collection or non-MongoDB exports?
I work on Dsync, a database migration tool, so weigh this accordingly.
mongoexport is a great tool for pulling one collection out to JSON or CSV. Two things I've wanted from it: the same kind of export from DynamoDB or Cosmos DB, and big exports that finish faster. What the MongoDB Database Tools docs say about its limits (mongoexport, mongodump):
- mongoexport exports one collection per run (
--collectionis required) and has no parallel option. The only way the docs show to split an export is--sortwith--skipand--limit, one command per piece. The usual workaround is parallel mongoexport processes, each with a--queryon an_idrange. The request for a multi-threaded mongoexport (TOOLS-248) was closed as Won't Fix. - JSON export is relaxed Extended JSON by default. Use
--jsonFormat=canonicalif you need every BSON type back.
If the data is only in MongoDB, a shell loop running mongoexport once per collection covers several collections.
For the other cases we added an open-source File connector to Dsync. It writes Dsync sources such as MongoDB and DynamoDB to CSV, one file per collection or table, and can load the files into another Dsync destination:
dsync --mode InitialSync --namespace "shop.orders,shop.customers" \
mongodb://localhost:27017 file:///data/export
On speed: once a collection has about 100,000 documents (twice the --doc-partition default of 50,000), Dsync splits it into _id ranges and reads them in parallel. The connector still writes its files one batch at a time, and we haven't formally benchmarked it against mongoexport.
We reworked how it handles values in v0.31.0. A key that first appears in a later document becomes a new column. ObjectIDs are written as hex, dates as ISO 8601, nested documents and arrays as Extended JSON, and on reload _id and most types come back.
Disclaimer on current limits: CSV only, no field selection. A null, a missing field and an empty string all reload as missing. A top-level string of digits, such as an order number, comes back as a number unless you pass --infer-types=false (digit strings with a leading zero stay strings), and a string holding a JSON object or array comes back as an embedded document or array. The MongoDB source refuses a collection whose smallest and largest _id have different BSON types. An export overwrites an existing file for the same collection, and there's no CDC. For backups I'd still use mongodump.
Source (Go, AGPL, PRs welcome): https://github.com/adiom-data/dsync/tree/main/connectors/file