LoginRadius is one of the leading and technologically advanced Customer Identity and Access Management (CIAM) solutions. Enterprise customers rely on our CIAM to manage end-user authentication and authorization. They typically serve hundreds of thousands to millions of end-users, making our CIAM a critical part of their IT infrastructure and value delivery.
\nOur backend consists of multiple microservices handling various identity and access management functions and workflows through APIs. And we use MongoDB as persistent storage for configuration data. For faster access and availability of this data, we deployed Redis in-memory cache through Redis Enterprise Cloud.
\nWe had our configuration cache set up in Redis Cloud. And to reduce the Redis Cloud latency, we kept the configuration cache at the application level in memory — but we ran into problems.
\nGenerally, customers don’t update their configurations so frequently. But when a customer updates their configuration, it doesn’t propagate in the backend until the server memory cache is purged — sometimes even taking several hours.
\nThis is bad for business: A customer updates configurations in response to a new requirement or rapidly changing business environment. If these changes take so much time for a digital identity process, it can affect end-users and, in turn, business outcomes. Simply imagine that a customer updated app configuration to accommodate a one-time flash sale, and end-users can’t place orders properly due to configuration update issues!
\nSo, we started evaluating various options to address these issues. We considered running multiple instances in the Redis Cloud and synchronizing them to minimize latency for all regions while ensuring customer configuration updates go live immediately. But this proved to be technically cumbersome and costly.
\nWe continued our research with various solutions and concluded that AWS ElastiCache for Redis best serves our needs.
\nAWS provides ElastiCache for Redis as a Redis Cloud alternative with all necessary capabilities. Also, we were already using AWS Cloud for some of our IT infrastructure needs.\nSo, we can deploy ElastiCache alongside the same infrastructure to solve the latency issues.
\nAccordingly, we have created ElastiCache instances in multiple AWS regions and set up the primary ElastiCache DB to quickly sync configuration updates in the secondary ElastiCache instances. Also, we deployed ElastiCache instances in multiple locations as needed.
\nFor migration, we updated the old Redis and ElastiCache primary instances simultaneously. Once we reached a sufficient confidence level with the new setup, we completely switched over to ElastiCache.
\nOur applications and cache are deployed in AWS, so our API response latency is no longer problematic. Ultimately, we can reduce application in memory cache updates to a few minutes or seconds as required.
\nNow customers get updated configurations deployed rapidly, solving our primary challenge!
\n","frontmatter":{"date":"August 09, 2023","updated_date":null,"title":"Breaking Down the Decision: Why We Chose AWS ElastiCache Over Redis Cloud","tags":["Cache","AWS","Redis","LoginRadius"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/f73d944745cd7f94227e1f20e4214968/14b42/migrating-to-aws-elasticache-for-redis.jpg","srcSet":"/static/f73d944745cd7f94227e1f20e4214968/f836f/migrating-to-aws-elasticache-for-redis.jpg 200w,\n/static/f73d944745cd7f94227e1f20e4214968/2244e/migrating-to-aws-elasticache-for-redis.jpg 400w,\n/static/f73d944745cd7f94227e1f20e4214968/14b42/migrating-to-aws-elasticache-for-redis.jpg 800w,\n/static/f73d944745cd7f94227e1f20e4214968/47498/migrating-to-aws-elasticache-for-redis.jpg 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Kundan Singh","github":null,"avatar":null}}}},{"node":{"fields":{"slug":"/engineering/ec2-instance-aws/"},"html":"AWS EC2 is a virtual computing environment (known as instances) to develop and deploy applications. To create an EC2 instance in AWS, we need an active Amazon Web Services account.
\nFirst, let's login into our AWS account. Once login, we will land on the Management Console page, we can see all the AWS services.
\nThere is a drop-down on the top left corner, which is one more option to search and browse all the services provided by AWS. You can find EC2 under the Compute category.
There is a button Launch Instance to create a new instance in the selected region on the dashboard's bottom section.
After clicking the Launch button, we need to select the Amazon Machine Image (AMI) it includes the operating system and applications required to launch an instance. Here we will select the Ubuntu Server 20.04 LTS as shown in the below snapshot.
\nWe can increase or decrease the size of instance storage while creating it; the free tier is eligible upto 30GB; if you need more storage, it will be billed according to Elastic Block Store (EBS) Pricing
\nOn the next screen, we can add tags to our instance and storage; these tags are key-value pair which are very useful to add properties to our resources, especially when we have multiple instances.
\nWe can define the firewall rules in a security group attached to our instance. With the help of these rules, we can control the traffic to our instance.
\nOnce we are done with firewall rules, we can review the complete detail of our new instance on a single page, and here we can click Launch button to launch the instance.
When we click Launch, it will open a pop-up that will require you to select a pre-existing public-private key-pair or create one to connect to our instance securely. Once you select/download the key, you will be able to launch the instance.
It will take approximately 5-10 min to launch the instance. Once launched, You can see the list of your running instances by clicking on the Instances button on the left menu.
Login to EC2 via CLI
\n$ ssh -i ec2-key.pem ec2-user@11.11.11.111Switch to ROOT
\nsudo suAmazon Relational Database Service is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides a resizable, cost-efficient capacity for an industry-standard relational database and manages everyday database administration tasks.\nAfter setting up RDS, you can be logged in to the local tool(SQL Developer/ MY SQL Workbench) and copy RDS's endpoint with Username/Password configured while setting RDS.
\nThis AWS service is used to Create, Publish, Maintain, and Monitor secure Application Programming Interfaces(API). Moreover, it provides an easy interface for code running on AWS Lambda.\nHere are some of the pointers that explain the need for this service.
\nThis service enables us to make APIs on the go with a few clicks; also, we can use the Mocking approach to make dummy routes without any original data. In this faking data is used and bouncing back to without any activity.
\nDynamo DB is a NO SQL database provided by Amazon Web Services. The main job of Dynamo DB is to Store & Retrieve any amount of data and serve any level requests of traffic.\nIn this, there are different terminologies as that of a regular database. Secondary Indexes is a Data Structure that contains a subset of attributes from a table.
\nLocal & Global Secondary Indexes
\n$0.0416/hour for the instance and additional pricing based on the EBS type and storage (~ calculator.aws)
\n\n\nIt is almost 4.5x savings just in terms of the infrastructure!
\n
Now that you know the major why(s) and are still reading this post, without further ado, let’s dive into the tech.
\nSince the entire content can be a bit exhausting in one place, I’m going to divide this into 2 related posts.
\nI’m going to mention below the guide for setting up an instance running RedHat Enterprise Linux 8 on AWS. This is because MongoDB generally performs better with the xfs file-system.
\nI’ve used a t3.small instance that comes with 2 vCPUs and 2Gb of RAM albeit you can select any instance of your choice.
\n\nIt is recommended that your DB should have access to at least 1GB of RAM and 2 real cores as that directly affects the performance during caching & concurrency mechanisms as handled by the default engine WiredTiger. You can read more about the production notes related to the RAM and CPU requirements here.
\n
Configuration Overview:
\nI’m assuming that you’re familiar with creating a VM on AWS.
\nNow, once the instance is in running state, assign an Elastic IP to it and then simply do a remote login into the machine.
\n\n\nWe'll need the Elastic IP to setup public hostname for the instance
\n
$ ssh -i <PEM_FILE> ec2-user@<ELASTIC_IP>We have added 2 additional EBS volumes other than the Root FS for Data and Logs which are yet to be mounted (Remember the 30Gb and 3Gb?). You can list the volume blocks using,
\n$ sudo lsblkThe additional volumes will be listed right after the root block (refer to the arrows)
\n$ sudo blkidCopy the UUIDs listed for /dev/xvdb and /dev/xvdc. Refer to the “LABEL” for block identification
\nNow open the /etc/fstab file and paste the configuration in the following format.
UUID=<COPIED_UUID_FOR_DATA> /var/lib/mongo xfs defaults,nofail 0 0\nUUID=<COPIED_UUID_FOR_LOGS> /var/log/mongodb xfs defaults,nofail 0 0The hostname will be used to identify your database server on the network. You can either use the above assigned Elastic IP or Domain name (if available). Open the /etc/hostname file and append the entry. For e.g.
ip-xx.us-east-2.compute.internal **<ELASTIC_IP> <DOMAIN_1> <DOMAIN_2>** ...This is optionally required in order to control the maximum number of acceptable connections while keeping the system stable.\nOpen the /etc/security/limits.conf file and add the following entries.
* soft nofile 64000\n* hard nofile 64000\n* soft nproc 32000\n* hard nproc 32000Now that all of the infra related prerequisites are sorted, reboot the instance, and let’s proceed to MongoDB installation.
\nCreate a file /etc/yum.repos.d/mongodb-org.4.2.repo and add the following package repository details.
[mongodb-org-4.2]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://www.mongodb.org/static/pgp/server-4.2.ascNow let’s install MongoDB.
\n$ sudo yum -y install mongodb-orgMongoDB by default uses the following paths to store the data and the internal logs:
\n\n\n/var/lib/mongo → Data
\n
/var/log/mongodb → Logs
Create the directories
\n$ sudo mkdir /var/lib/mongo\n$ sudo mkdir /var/log/mongodbChange user & group permissions
\n$ sudo chown mongod:mongod /var/lib/mongo\n$ sudo chown mongod:mongod /var/log/mongodThe mongod daemon/service must be first running before we proceed to create a user.\nLet’s use the default config(stored in /etc/mongod.conf) for now and start the daemon process.
$ sudo -u mongod mongod -f /etc/mongod.confThe above command will start the mongod daemon in fork mode (default).
\nNow, let’s login to the server and create our first admin user.
\nOpen a mongo shell
\n$ mongoUse the \"admin\" database to create the root-admin
\n> use adminCreate the admin user
\n> db.createUser({user: "admin", pwd: "password", roles: [{role: "root", db: "admin"}]})Create a regular user
\n> db.createUser({user: "normal_user", pwd: "password", roles: [{role: "readWriteAnyDatabase", db: "admin"}]})Shutdown the server for now. We'll restart again with the modified config
\n> db.shutDownServer()Here, we’ll enable the database authentication and modify the bind-address for our server to be accessible in the public domain.\nOpen /etc/mongod.conf and make the below changes.
# network interfaces\nnet:\n port: 27017\n bindIp: 0.0.0.0 # accessible on the network address\nsecurity:\n authorization: enabled # creds will be required for making db operationsSave the config and restart the server.
\n$ sudo -u mongod mongod -f /etc/mongod.confYou can verify if the credentials work using,
\n$ mongo -u admin -p passwordThat is it about the initial setup! Please stay tuned for my next related post on the detailed migration process and tips on keeping the DB production-ready.
\nP.S. Thanks to Piyush Kumar for helping curate this post!
\n","frontmatter":{"date":"June 30, 2020","updated_date":null,"title":"Self-Hosted MongoDB","tags":["MongoDB","Mongo","AWS","Atlas"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":2.0408163265306123,"src":"/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg","srcSet":"/static/504642ca3a1f7d78dea8509436faa4c6/f836f/cover.jpg 200w,\n/static/504642ca3a1f7d78dea8509436faa4c6/2244e/cover.jpg 400w,\n/static/504642ca3a1f7d78dea8509436faa4c6/14b42/cover.jpg 800w,\n/static/504642ca3a1f7d78dea8509436faa4c6/d8343/cover.jpg 925w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Chinmaya Pati","github":"cnp96","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/failover-systems-and-loginradius-99-99-uptime/"},"html":"Do you remember the Amazon outage that affected several high-profile customers back in 2011? On April 21st, widely-used sites such as Reddit and Quora were brought down, and many others experienced latency or were knocked offline, too. Early that morning, as part of normal scaling activities, Amazon staff performed a network change. However, the change was done incorrectly and, after the staff attempted to correct it through a rollback, the inner mechanisms of the service made it unavailable to serve read and write requests. The ‘inner mechanisms’ responsible for the service unavailability are out of the scope of this article, but they are described in the service disruption summary that Amazon published. What interests us here is the strategies that companies can use to mitigate the negative effects on their products when a service they rely on fails. These strategies make up what is known as a failover system.
\nA failover system is a set of mechanisms that perform failover. In computer networking, failover is the process of switching to a redundant or standby server or network upon the abnormal termination of a previously-active server or network. The benefit of a failover system is obvious: with one in place, you can ensure your product or service will be available even when adverse events take place. Uptime—time during which a service is operational—is crucial to the success of your business. If your services are unavailable, you are likely to lose customers and any revenue they would have generated. In an article entitled ‘Lessons Netflix Learned from the AWS Outage,’ Netflix talks about the manual process they used to deal with the Amazon outage and acknowledges the importance of automating this process in the future so they can keep scaling their service. That is, the company acknowledges the importance of having an automated failover system in place instead of relying on a team of top engineers manually making changes every time there is an issue.
\nHow is an automated failover system set up? Let us consider the simplest scenario: setting a failover system for one server. By definition of a failover system, in addition to the main server, we need to have another redundant standby server that we can switch to whenever the main one fails. Since the redundant server must provide the same functionality as the first, it must be identical to it. Additionally, we need a tool that ensures client requests are routed to the redundant server in case of failure of the main one. We can achieve this by using a DNS failover tool. DNS is the internet protocol used to translate human-readable hostnames into IP addresses, and a DNS failover tool makes sure the “dictionary” (DNS tables) used for this translation are updated in the event of an outage. DNS failover tools know when to update the tables by periodically checking on the main server’s status. With these three tools—and some configuration—you can set up a simple, automated failover system. Of course, there are other considerations you must take when setting the failover system, such as ensuring the redundant standby server is hosted in a geographic area different from the main server’s location and using different companies to host your services. That way, your services will be less likely to go down simultaneously.
\nAt LoginRadius, we have set up automated failover systems in all layers of our architecture, which is why we can ensure 99.99% uptime on a monthly basis. With our services, you can be assured your customers will always be able to engage with your business, ensuring you will never lose these customers and any revenue they will generate.
\n","frontmatter":{"date":"June 24, 2019","updated_date":null,"title":"Failover Systems and LoginRadius' 99.99% Uptime","tags":["Engineering","AWS"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.0050251256281406,"src":"/static/b25cc95f0b8e5c745ef8652dfb386e3b/40823/tech.jpg","srcSet":"/static/b25cc95f0b8e5c745ef8652dfb386e3b/f836f/tech.jpg 200w,\n/static/b25cc95f0b8e5c745ef8652dfb386e3b/2244e/tech.jpg 400w,\n/static/b25cc95f0b8e5c745ef8652dfb386e3b/40823/tech.jpg 719w","sizes":"(max-width: 719px) 100vw, 719px"}}},"author":{"id":"Ruben Gonzalez","github":"rubenprograms","avatar":null}}}}]}},"pageContext":{"tag":"AWS"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}