{"componentChunkName":"component---src-pages-author-author-yaml-id-js","path":"/author/chinmaya-pati/","result":{"data":{"allMarkdownRemark":{"edges":[{"node":{"id":"8d1c6c7b-3313-5137-89fa-d096a463287e","html":"

The goal of this post is to learn about the various ways of data migration in MongoDB that can help us to write scripts that change your database by adding new documents, modifying existing ones.

\n

If you're coming here for the first time, please take a look at the prequel Self-Hosted MongoDB.

\n

Alright then, picking from where we left off, let's get started with the data migration in MongoDB.

\n

Now, the basic steps to migrate data from one MongoDB to another would be:

\n
    \n
  1. Create a zipped backup of the existing data
    2. \n
  2. Dump the data in a new DB
    3. \n
\n

This is very straight forward when the source database is not online because we know that there won't be any new documents created/updated during the migration process.\nLet's look at simple migration first before diving into the live scenario.

\n
\n

Migrating from an offline database in MongoDB

\n

Creating a backup

\n

We're going to use an existing utility program mongodump for creating the database backup.

\n

Run this command in the source database server

\n
mongodump --host="hostname:port" \\\n  --username="username" --password="password" \\\n  --authenticationDatabase "admin" \\\n  --db="db name" --collection="collection name" --query='json' \\\n  --forceTableScan -v --gzip --out ./dump
\n

--host: The source MongoDB hostname along with the port. It defaults to localhost:27017. If it is a connection string you can use this option —-uri=\"mongodb://username:password@host1[:port1]...\"

\n

--username: Specifies a username to authenticate to a MongoDB database that uses authentication.

\n

--password: Specifies a password to authenticate to a MongoDB database that uses authentication.

\n

--authenticationDatabase: Specifies the authentication database where the specified --username has been created.

\n
\n

If you do not specify an authentication database or a database to export, mongodump assumes the admin database holds the user's credentials.

\n
\n

--db: Specifies the database to take a backup from. If you do not specify a database, mongodump collects from all databases in this instance.

\n
\n

Alternatively, you can also specify the database directly in the URI connection string i.e. mongodb://username:password@uri/dbname.
Providing a connection string while also using --db and specifying conflicting information will result in an error.

\n
\n

--collection: Specifies a collection to backup. If you do not specify a collection, this option copies all collections in the specified database or instance to the dump files.

\n

--query : Provides a JSON document as a query that optionally limits the documents included in the output of mongodump.
\nYou must enclose the query document in single quotes ('{ ... }') to ensure that it does not interact with your environment.
\nThe query must be in Extended JSON v2 format (either relaxed or canonical/strict mode), including enclosing the field names and operators in quotes e.g. '{ \"created_at\": { \"\\$gte\": ISODate(...) } }'.

\n
\n

To use the --query option, you must also specify the --collection option.

\n
\n

--forceTableScan: Forces mongodump to scan the data store directly. Typically, mongodump saves entries as they appear in the index of the _id field.

\n
\n

If you specify a query --query, mongodump will use the most appropriate index to support that query.
Hence , you cannot use --forceTableScan with the --query option.

\n
\n

--gzip: Compresses the output. If mongodump outputs to the dump directory, the new feature compresses the individual files. The files have the suffix .gz.

\n

--out: Specifies the directory where mongodump will write BSON files for the dumped databases. By default, mongodump saves output files in a directory named dump in the current working directory.

\n

Restoring the backup

\n

We will use a utility program called mongorestore for restoring the database backup.

\n

Copy the backup directory dump to the new Database instance and run the following command:

\n
mongorestore --uri="mongodb://user:password@host:port/?authSource=admin" \\\n  --drop --noIndexRestore --gzip -v ./dump
\n

Replace the credentials with the new database credentials. Unline in the previous step, the --authenticationDatabase option is specified in the URI string.

\n

Also, use --gzip if used while creating the backup.

\n

--drop: Before restoring the collections from the dumped backup, drops the collections from the target database. It does not drop collections that are not in the backup.\n--noIndexRestore: Prevents mongorestore from restoring and building indexes as specified in the corresponding mongodump output.

\n
\n

If you want to change name of the database while restoring, you can do so using
--nsFrom=\"old_name.*\" --nsTo=\"new_name.*\" options.

However, it won’t work if you were to migrate with oplogs which is a requirement in migration from an online instance.

\n
\n
\n

Migrating from an online database in MongoDB

\n

The only challenge with migrating from an online database is not able to pause the updates during migration. So here is the overview of the steps,

\n
    \n
  1. Run an initial bulk migration with oplogs capture
    2. \n
  2. Run a sync job to mitigate the database connection switch latency
    3. \n
\n
\n

Now, to capture oplogs, a replica set must be initialized in the source and destination databases. This is because the oplogs are captured from local.oplog.rs namespace, which is created after initializing a replica set.

You can follow this guide to configure a replica set.

\n
\n

Initial Migration with Oplog Capture

\n

Oplogs, in simple words, are the operation logs created per operation in the database. They represent a partial document state or, in other words, the database state. So we are going to capture any updates in our old database during the migration process using these oplogs.

\n

Run the mongodump program with the following options,

\n
mongodump --uri=".../?authSource=admin" \\\n  --forceTableScan --oplog \\\n  --gzip -v --out ./dump
\n

--oplog: Creates a file named oplog.bson as part of the mongodump output. The oplog.bson file, located in the top level of the output directory, contains oplog entries that occur during the mongodump operation. This file provides an effective point-in-time snapshot of the state of our database instance.

\n

Restore the data with oplog replay

\n

In order to replay the oplogs, a special role is required. Let's create and assign the role to the database user being used for migration.

\n

Create the role

\n
db.createRole({\n  role: "interalUseOnlyOplogRestore",\n  privileges: [\n    {\n      resource: { anyResource: true },\n      actions: [ "anyAction" ] \n    }\n  ],\n  roles: []\n})
\n

Assign the role

\n
db.grantRolesToUser(\n  "admin",\n  [{ role:"interalUseOnlyOplogRestore", db:"admin" }]\n);
\n

Now you can restore using the mongorestore program with the following options,

\n
mongorestore --uri="mongodb://admin:.../?authSource=admin" \\\n  --oplogReplay \n  --gzip -v ./dump
\n

In the above command, using the same user admin with whom the role was associated.

\n

--oplogReplay: After restoring the database dump, replays the oplog entries from a bson file and restores the database to the point-in-time backup captured with the mongodump --oplog command.

\n

Mitigating database connection switch latency

\n

Alright, so far we are done with most of the heavy lifting. The only thing that remains is maintaining consistency between the databases during the connection switch in our application servers.

\n
\n

If you're running MongoDB version 3.6+, it's better to go for the Change Stream approach, which is a event-based mechanism introduced to capture changes in your database in an optimized way. Here is an article that covers it : An Introduction to Change Streams

\n
\n

Check out the generic sync script, which you can run as a CRON job every minute.

\n

Update the variables in this script and run as

\n
$ ./delta-sync.sh from_epoch_in_milliseconds\n\n# from_epoch_in_milliseconds is automatically picked with every iteration if not supplied
\n

Or you can set up a cron job to run this every minute.

\n
* * * * * ~/delta-sync.sh
\n

The output can be monitored with the following command (I'm running RHEL 8, refer to your OS guide for cron output)

\n
$ tail -f /var/log/cron | grep CRON
\n

This is a sample sync log.

\n
CMD (~/cron/dsync.sh)\nCMDOUT (INFO: Updated log registry to use new timestamp on next run.)\nCMDOUT (INFO: Created sync directory: /home/ec2-user/cron/dump/2020-11-03T19:01:01Z)\nCMDOUT (Fetching oplog in range [2020-11-03T19:00:01Z - 2020-11-03T19:01:01Z])\nCMDOUT (2020-11-03T19:01:02.319+0000#011dumping up to 1 collections in parallel)\nCMDOUT (2020-11-03T19:01:02.334+0000#011writing local.oplog.rs to /home/ec2-user/cron/dump/2020-11-03T19:01:01Z/local/oplog.rs.bson.gz)\nCMDOUT (2020-11-03T19:01:04.943+0000#011local.oplog.rs  0)\nCMDOUT (2020-11-03T19:01:04.964+0000#011local.oplog.rs  0)\nCMDOUT (2020-11-03T19:01:04.964+0000#011done dumping local.oplog.rs (0 documents))\nCMDOUT (INFO: Dump success!)\nCMDOUT (INFO: Replaying oplogs...)\nCMDOUT (2020-11-03T19:01:05.030+0000#011using write concern: &{majority false 0})\nCMDOUT (2020-11-03T19:01:05.054+0000#011will listen for SIGTERM, SIGINT, and SIGKILL)\nCMDOUT (2020-11-03T19:01:05.055+0000#011connected to node type: standalone)\nCMDOUT (2020-11-03T19:01:05.055+0000#011mongorestore target is a directory, not a file)\nCMDOUT (2020-11-03T19:01:05.055+0000#011preparing collections to restore from)\nCMDOUT (2020-11-03T19:01:05.055+0000#011found collection local.oplog.rs bson to restore to local.oplog.rs)\nCMDOUT (2020-11-03T19:01:05.055+0000#011found collection metadata from local.oplog.rs to restore to local.oplog.rs)\nCMDOUT (2020-11-03T19:01:05.055+0000#011restoring up to 4 collections in parallel)\nCMDOUT (2020-11-03T19:01:05.055+0000#011replaying oplog)\nCMDOUT (2020-11-03T19:01:05.055+0000#011applied 0 oplog entries)\nCMDOUT (2020-11-03T19:01:05.055+0000#0110 document(s) restored successfully. 0 document(s) failed to restore.)\nCMDOUT (INFO: Restore success!)
\n

You can stop this script after verifying that no more oplogs are being created, i.e., when source DB went offline.

\n

This concludes the complete self-hosted MongoDB data migration guide. If you want to learn more about MongoDB here is a useful resource on how to use MongoDB as datasource in goLang.

\n","frontmatter":{"title":"How to Migrate Data In MongoDB","author":{"id":"Chinmaya Pati","github":"cnp96","avatar":null},"date":"December 14, 2020","updated_date":null,"tags":["MongoDB"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.4184397163120568,"src":"/static/90f92104129cbc152fa1d9a64c58d7be/ee604/index.png","srcSet":"/static/90f92104129cbc152fa1d9a64c58d7be/69585/index.png 200w,\n/static/90f92104129cbc152fa1d9a64c58d7be/497c6/index.png 400w,\n/static/90f92104129cbc152fa1d9a64c58d7be/ee604/index.png 800w,\n/static/90f92104129cbc152fa1d9a64c58d7be/31987/index.png 1000w","sizes":"(max-width: 800px) 100vw, 800px"}}}},"fields":{"authorId":"Chinmaya Pati","slug":"/engineering/live-data-migration-mongodb/"}}},{"node":{"id":"abd9eda6-e048-505e-acf5-f578d763cf94","html":"

You’re probably hosting your MongoDB on a reliable cloud service provider say Atlas for instance because you really want to focus on your idea and delegate all the subtle key management areas such as networking, storage, access, etc.

\n

It all looks good initially until your small idea starts turning into a business and the cost starts skyrocketing. Even if that is not the case, this post will still give you a general overview of the technical complexities involved (and bucks saved!) if you were to migrate to a self-hosted solution.

\n

BTW, how much savings are we talking about? Let’s do a quick comparison between an Atlas instance and a self-hosted MongoDB on AWS.

\n

Atlas (~$166/month)

\n

\n \n \n

\n

$0.0416/hour for the instance and additional pricing based on the EBS type and storage (~ calculator.aws)

\n
\n

It is almost 4.5x savings just in terms of the infrastructure!

\n
\n

Now that you know the major why(s) and are still reading this post, without further ado, let’s dive into the tech.

\n
\n

Outline

\n
    \n
  1. Setting up the infrastructure
    2. \n
  2. Setting up MongoDB
    3. \n
  3. Bulk migration
    4. \n
  4. Delta-sync to bridge the connection switch latency (not applicable to stale clusters)
    5. \n
\n

Since the entire content can be a bit exhausting in one place, I’m going to divide this into 2 related posts.

\n

1. Setting up the Infrastructure

\n

I’m going to mention below the guide for setting up an instance running RedHat Enterprise Linux 8 on AWS. This is because MongoDB generally performs better with the xfs file-system.

\n

Spin up an EC2 Instance

\n

I’ve used a t3.small instance that comes with 2 vCPUs and 2Gb of RAM albeit you can select any instance of your choice.

\n
\n

It is recommended that your DB should have access to at least 1GB of RAM and 2 real cores as that directly affects the performance during caching & concurrency mechanisms as handled by the default engine WiredTiger. You can read more about the production notes related to the RAM and CPU requirements here.

\n
\n

Configuration Overview:

\n\n

I’m assuming that you’re familiar with creating a VM on AWS.

\n

Now, once the instance is in running state, assign an Elastic IP to it and then simply do a remote login into the machine.

\n
\n

We'll need the Elastic IP to setup public hostname for the instance

\n
\n
$ ssh -i <PEM_FILE> ec2-user@<ELASTIC_IP>
\n

Mount Additional Volumes

\n

We have added 2 additional EBS volumes other than the Root FS for Data and Logs which are yet to be mounted (Remember the 30Gb and 3Gb?). You can list the volume blocks using,

\n
$ sudo lsblk
\n

The additional volumes will be listed right after the root block (refer to the arrows)

\n

\n because they are unique and won't change over a system restart)

\n
$ sudo blkid
\n

\n \n \n

\n

Copy the UUIDs listed for /dev/xvdb and /dev/xvdc. Refer to the “LABEL” for block identification

\n

Now open the /etc/fstab file and paste the configuration in the following format.

\n
UUID=<COPIED_UUID_FOR_DATA> /var/lib/mongo xfs defaults,nofail 0 0\nUUID=<COPIED_UUID_FOR_LOGS> /var/log/mongodb xfs defaults,nofail 0 0
\n

Update Hostname

\n

The hostname will be used to identify your database server on the network. You can either use the above assigned Elastic IP or Domain name (if available). Open the /etc/hostname file and append the entry. For e.g.

\n
ip-xx.us-east-2.compute.internal **<ELASTIC_IP> <DOMAIN_1> <DOMAIN_2>** ...
\n

Update the Process Limits (Optional)

\n

This is optionally required in order to control the maximum number of acceptable connections while keeping the system stable.\nOpen the /etc/security/limits.conf file and add the following entries.

\n
* soft nofile 64000\n* hard nofile 64000\n* soft nproc 32000\n* hard nproc 32000
\n

Now that all of the infra related prerequisites are sorted, reboot the instance, and let’s proceed to MongoDB installation.

\n
\n

1. Setting up MongoDB

\n

Add the Repo Source

\n

Create a file /etc/yum.repos.d/mongodb-org.4.2.repo and add the following package repository details.

\n
[mongodb-org-4.2]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
\n

Now let’s install MongoDB.

\n
$ sudo yum -y install mongodb-org
\n

Create directories and setup permissions

\n

MongoDB by default uses the following paths to store the data and the internal logs:

\n
\n

/var/lib/mongo → Data
/var/log/mongodb → Logs

\n
\n

Create the directories

\n
$ sudo mkdir /var/lib/mongo\n$ sudo mkdir /var/log/mongodb
\n

Change user & group permissions

\n
$ sudo chown mongod:mongod /var/lib/mongo\n$ sudo chown mongod:mongod /var/log/mongod
\n

Create an Admin User

\n

The mongod daemon/service must be first running before we proceed to create a user.\nLet’s use the default config(stored in /etc/mongod.conf) for now and start the daemon process.

\n
$ sudo -u mongod mongod -f /etc/mongod.conf
\n

The above command will start the mongod daemon in fork mode (default).

\n

Now, let’s login to the server and create our first admin user.

\n

Open a mongo shell

\n
$ mongo
\n

Use the \"admin\" database to create the root-admin

\n
> use admin
\n

Create the admin user

\n
> db.createUser({user: "admin", pwd: "password", roles: [{role: "root", db: "admin"}]})
\n

Create a regular user

\n
> db.createUser({user: "normal_user", pwd: "password", roles: [{role: "readWriteAnyDatabase", db: "admin"}]})
\n

Shutdown the server for now. We'll restart again with the modified config

\n
> db.shutDownServer()
\n

Setting up Authentication

\n

Here, we’ll enable the database authentication and modify the bind-address for our server to be accessible in the public domain.\nOpen /etc/mongod.conf and make the below changes.

\n
# network interfaces\nnet:\n  port: 27017\n  bindIp: 0.0.0.0 # accessible on the network address\nsecurity:\n  authorization: enabled # creds will be required for making db operations
\n

Save the config and restart the server.

\n
$ sudo -u mongod mongod -f /etc/mongod.conf
\n

Test Login

\n

You can verify if the credentials work using,

\n
$ mongo -u admin -p password
\n

That is it about the initial setup! Please stay tuned for my next related post on the detailed migration process and tips on keeping the DB production-ready.

\n

P.S. Thanks to Piyush Kumar for helping curate this post!

\n","frontmatter":{"title":"Self-Hosted MongoDB","author":{"id":"Chinmaya Pati","github":"cnp96","avatar":null},"date":"June 30, 2020","updated_date":null,"tags":["MongoDB","Mongo","AWS","Atlas"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":2.0408163265306123,"src":"/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg","srcSet":"/static/504642ca3a1f7d78dea8509436faa4c6/f836f/cover.jpg 200w,\n/static/504642ca3a1f7d78dea8509436faa4c6/2244e/cover.jpg 400w,\n/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg 800w,\n/static/504642ca3a1f7d78dea8509436faa4c6/d8343/cover.jpg 925w","sizes":"(max-width: 800px) 100vw, 800px"}}}},"fields":{"authorId":"Chinmaya Pati","slug":"/engineering/self-hosted-mongo/"}}}]},"authorYaml":{"id":"Chinmaya Pati","bio":"I'm an avid FOSS enthusiast and contributor interested in system design, web-dev, UI/UX, data-driven technologies, and DevOps.","github":"cnp96","stackoverflow":"5790355","linkedin":"chinmayapati","medium":"@chinmaya-cp","twitter":"chiku__p","avatar":null}},"pageContext":{"id":"Chinmaya Pati","__params":{"id":"chinmaya-pati"}}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}