{"componentChunkName":"component---src-templates-tag-js","path":"/tags/mongo/","result":{"data":{"site":{"siteMetadata":{"title":"LoginRadius Blog"}},"allMarkdownRemark":{"totalCount":2,"edges":[{"node":{"fields":{"slug":"/engineering/self-hosted-mongo/"},"html":"

You’re probably hosting your MongoDB on a reliable cloud service provider say Atlas for instance because you really want to focus on your idea and delegate all the subtle key management areas such as networking, storage, access, etc.

\n

It all looks good initially until your small idea starts turning into a business and the cost starts skyrocketing. Even if that is not the case, this post will still give you a general overview of the technical complexities involved (and bucks saved!) if you were to migrate to a self-hosted solution.

\n

BTW, how much savings are we talking about? Let’s do a quick comparison between an Atlas instance and a self-hosted MongoDB on AWS.

\n

Atlas (~$166/month)

\n

\n \n \n

\n

$0.0416/hour for the instance and additional pricing based on the EBS type and storage (~ calculator.aws)

\n
\n

It is almost 4.5x savings just in terms of the infrastructure!

\n
\n

Now that you know the major why(s) and are still reading this post, without further ado, let’s dive into the tech.

\n
\n

Outline

\n
    \n
  1. Setting up the infrastructure
    2. \n
  2. Setting up MongoDB
    3. \n
  3. Bulk migration
    4. \n
  4. Delta-sync to bridge the connection switch latency (not applicable to stale clusters)
    5. \n
\n

Since the entire content can be a bit exhausting in one place, I’m going to divide this into 2 related posts.

\n

1. Setting up the Infrastructure

\n

I’m going to mention below the guide for setting up an instance running RedHat Enterprise Linux 8 on AWS. This is because MongoDB generally performs better with the xfs file-system.

\n

Spin up an EC2 Instance

\n

I’ve used a t3.small instance that comes with 2 vCPUs and 2Gb of RAM albeit you can select any instance of your choice.

\n
\n

It is recommended that your DB should have access to at least 1GB of RAM and 2 real cores as that directly affects the performance during caching & concurrency mechanisms as handled by the default engine WiredTiger. You can read more about the production notes related to the RAM and CPU requirements here.

\n
\n

Configuration Overview:

\n\n

I’m assuming that you’re familiar with creating a VM on AWS.

\n

Now, once the instance is in running state, assign an Elastic IP to it and then simply do a remote login into the machine.

\n
\n

We'll need the Elastic IP to setup public hostname for the instance

\n
\n
$ ssh -i <PEM_FILE> ec2-user@<ELASTIC_IP>
\n

Mount Additional Volumes

\n

We have added 2 additional EBS volumes other than the Root FS for Data and Logs which are yet to be mounted (Remember the 30Gb and 3Gb?). You can list the volume blocks using,

\n
$ sudo lsblk
\n

The additional volumes will be listed right after the root block (refer to the arrows)

\n

\n because they are unique and won't change over a system restart)

\n
$ sudo blkid
\n

\n \n \n

\n

Copy the UUIDs listed for /dev/xvdb and /dev/xvdc. Refer to the “LABEL” for block identification

\n

Now open the /etc/fstab file and paste the configuration in the following format.

\n
UUID=<COPIED_UUID_FOR_DATA> /var/lib/mongo xfs defaults,nofail 0 0\nUUID=<COPIED_UUID_FOR_LOGS> /var/log/mongodb xfs defaults,nofail 0 0
\n

Update Hostname

\n

The hostname will be used to identify your database server on the network. You can either use the above assigned Elastic IP or Domain name (if available). Open the /etc/hostname file and append the entry. For e.g.

\n
ip-xx.us-east-2.compute.internal **<ELASTIC_IP> <DOMAIN_1> <DOMAIN_2>** ...
\n

Update the Process Limits (Optional)

\n

This is optionally required in order to control the maximum number of acceptable connections while keeping the system stable.\nOpen the /etc/security/limits.conf file and add the following entries.

\n
* soft nofile 64000\n* hard nofile 64000\n* soft nproc 32000\n* hard nproc 32000
\n

Now that all of the infra related prerequisites are sorted, reboot the instance, and let’s proceed to MongoDB installation.

\n
\n

1. Setting up MongoDB

\n

Add the Repo Source

\n

Create a file /etc/yum.repos.d/mongodb-org.4.2.repo and add the following package repository details.

\n
[mongodb-org-4.2]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
\n

Now let’s install MongoDB.

\n
$ sudo yum -y install mongodb-org
\n

Create directories and setup permissions

\n

MongoDB by default uses the following paths to store the data and the internal logs:

\n
\n

/var/lib/mongo → Data
/var/log/mongodb → Logs

\n
\n

Create the directories

\n
$ sudo mkdir /var/lib/mongo\n$ sudo mkdir /var/log/mongodb
\n

Change user & group permissions

\n
$ sudo chown mongod:mongod /var/lib/mongo\n$ sudo chown mongod:mongod /var/log/mongod
\n

Create an Admin User

\n

The mongod daemon/service must be first running before we proceed to create a user.\nLet’s use the default config(stored in /etc/mongod.conf) for now and start the daemon process.

\n
$ sudo -u mongod mongod -f /etc/mongod.conf
\n

The above command will start the mongod daemon in fork mode (default).

\n

Now, let’s login to the server and create our first admin user.

\n

Open a mongo shell

\n
$ mongo
\n

Use the \"admin\" database to create the root-admin

\n
> use admin
\n

Create the admin user

\n
> db.createUser({user: "admin", pwd: "password", roles: [{role: "root", db: "admin"}]})
\n

Create a regular user

\n
> db.createUser({user: "normal_user", pwd: "password", roles: [{role: "readWriteAnyDatabase", db: "admin"}]})
\n

Shutdown the server for now. We'll restart again with the modified config

\n
> db.shutDownServer()
\n

Setting up Authentication

\n

Here, we’ll enable the database authentication and modify the bind-address for our server to be accessible in the public domain.\nOpen /etc/mongod.conf and make the below changes.

\n
# network interfaces\nnet:\n  port: 27017\n  bindIp: 0.0.0.0 # accessible on the network address\nsecurity:\n  authorization: enabled # creds will be required for making db operations
\n

Save the config and restart the server.

\n
$ sudo -u mongod mongod -f /etc/mongod.conf
\n

Test Login

\n

You can verify if the credentials work using,

\n
$ mongo -u admin -p password
\n

That is it about the initial setup! Please stay tuned for my next related post on the detailed migration process and tips on keeping the DB production-ready.

\n

P.S. Thanks to Piyush Kumar for helping curate this post!

\n","frontmatter":{"date":"June 30, 2020","updated_date":null,"title":"Self-Hosted MongoDB","tags":["MongoDB","Mongo","AWS","Atlas"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":2.0408163265306123,"src":"/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg","srcSet":"/static/504642ca3a1f7d78dea8509436faa4c6/f836f/cover.jpg 200w,\n/static/504642ca3a1f7d78dea8509436faa4c6/2244e/cover.jpg 400w,\n/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg 800w,\n/static/504642ca3a1f7d78dea8509436faa4c6/d8343/cover.jpg 925w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Chinmaya Pati","github":"cnp96","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/guid-query-mongo-shell/"},"html":"

If you have stored a GUID through the C# driver to mongoDB and now you want to run a query by GUID, you can't query directly because mongoDB doesn't recognize GUID so when we query through mongo shell no result will be returned. To use the power of mongo shell for querying data on mongo by GUID, you should follow these steps.

\n

1. Convert GUID data to Base64

\n

Convert you GUID data to base64 , you can use any online tool for this.

\n

So suppose your GUID is: 00112233-4455-6677-8899-aabbccddeeff

\n

Then the base 64 version will be: MyIRAFVEd2aImaq7zN3u/w==

\n

2. Query by BinData object in mongo shell

\n
db.Users.find({"useUniqueId": new BinData(3,"MyIRAFVEd2aImaq7zN3u/w==")}).limit(1)
\n

Actually BinData constructor takes 2 parameters:

\n

New BinData(subtype,data)

\n\n

mongoDB's C# driver stores data to mongo by converting it into binary data rather than string.

\n","frontmatter":{"date":"February 23, 2015","updated_date":null,"title":"GUID Query Through Mongo Shell","tags":["Engineering","GUID","Mongo"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1,"src":"/static/ea7595df20b2f7c29ed276dc27b05c9b/630fb/mongo-db-guid.png","srcSet":"/static/ea7595df20b2f7c29ed276dc27b05c9b/69585/mongo-db-guid.png 200w,\n/static/ea7595df20b2f7c29ed276dc27b05c9b/630fb/mongo-db-guid.png 300w","sizes":"(max-width: 300px) 100vw, 300px"}}},"author":{"id":"Kundan Singh","github":null,"avatar":null}}}}]}},"pageContext":{"tag":"Mongo"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}