HTTP (hypertext transfer protocol) is a communication protocol that transfers data between client and server. HTTP requests are very essential to access resources from the same or remote server. In Golang, the net/http package comes with the default settings that we need to adjust according to our high-performance requirement.
For setting up HTTP clients for making requests, most programming languages have different frameworks in place. We will take a hands-on approach in the coming sections to explore how HTTP requests can be made in Golang or Go, as I will refer to the language for the rest of the post.
\nWhile working on the Golang projects, I realized that improper configuration of HTTP might crash your server anytime.
\nIn the time when I was working with HTTP Client, I Observed some problems and their solutions, listed below:
\nThe HTTP client does not contain the request timeout setting by default.\nIf you are using http.Get(URL) or &Client{} that uses the http.DefaultClient. DefaultClient has not timeout setting; it comes with no timeout
Suppose the Rest API where you are making the request is broken, not sending the response back that keeps the connection open. More requests came, and open connection count will increase, Increasing server resources utilization, resulting in crashing your server when resource limits are reached.
\nvar httpClient = &http.Client{\n Timeout: time.Second * 10,\n}For the Rest API, it is recommended that timeout should not more than 10 seconds.\nIf the Requested resource is not responded to in 10 seconds, the HTTP connection will be canceled with net/http: request canceled (Client.Timeout exceeded ...) error.
\nBy default, the Golang Http client performs the connection pooling. When the request completes, that connection remains open until the idle connection timeout (default is 90 seconds). If another request came, that uses the same established connection instead of creating a new connection, after the idle connection time, the connection will return to the pool.
\nUsing the connection pooling will keep less connection open and more requests will be served with minimal server resources.
\nWhen we not defined transport in the http.Client, it uses the default transport Go HTTP Transport
\nDefault configuration of the HTTP Transport,
\nvar DefaultTransport RoundTripper = &Transport{\n\t...\n\tMaxIdleConns: 100,\n\tIdleConnTimeout: 90 * time.Second,\n\t...\n}\n\nconst DefaultMaxIdleConnsPerHost = 2MaxIdleConns is the connection pool size, and this is the maximum connection that can be open; its default value is 100 connections.
\nThere is problem with the default setting DefaultMaxIdleConnsPerHost with value of 2 connection,\nDefaultMaxIdleConnsPerHost is the number of connection can be allowed to open per host basic.\nMeans for any particular host out of 100 connection from the connection pool only two connection will be allocated to that host.
With the more request came, it will process only two requests; other requests will wait for the connection to communicate with the host server and go in the TIME_WAIT state. As more request came, increase the connection to the TIME_WAIT state and increase the server resource utilization; at the limit, the server will crash.
t := http.DefaultTransport.(*http.Transport).Clone()\nt.MaxIdleConns = 100\nt.MaxConnsPerHost = 100\nt.MaxIdleConnsPerHost = 100\n\t\nhttpClient = &http.Client{\n Timeout: 10 * time.Second,\n Transport: t,\n}By increasing connection per host and the total number of idle connection, this will increase the performance and serve more request with minimal server resources.
\nConnection pool size and connection per host count can be increased as per server resources and requirements.
\nIn this article, we discussed the problems around the 'net/http' client default configurations. By changing some of the default settings of HTTP Client, we can achieve a High-performance HTTP client for production use. If you want to learn more about http, here is an interesting post on HTTP security headers If you like what you read, share your thoughts in the comment section.
\n","frontmatter":{"date":"January 12, 2021","updated_date":null,"description":"In this post, we will discuss how to make HTTP requests for higher performance in Go, and also how to tune it.","title":"How to Use the HTTP Client in GO To Enhance Performance","tags":["Golang","HTTP","Performance"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.7699115044247788,"src":"/static/68276d4b4277ad09e329598038e91917/14b42/index.jpg","srcSet":"/static/68276d4b4277ad09e329598038e91917/f836f/index.jpg 200w,\n/static/68276d4b4277ad09e329598038e91917/2244e/index.jpg 400w,\n/static/68276d4b4277ad09e329598038e91917/14b42/index.jpg 800w,\n/static/68276d4b4277ad09e329598038e91917/47498/index.jpg 1200w,\n/static/68276d4b4277ad09e329598038e91917/ec6c5/index.jpg 1280w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Mayank Agarwal","github":"mayankagrwal","avatar":null}}}},{"node":{"excerpt":"One fairly popular question that got asked on programming bulletin boards has to do with the similarities and differences between React’s…","fields":{"slug":"/engineering/react-constructor-get-initial-state/"},"html":"One fairly popular question that got asked on programming bulletin boards has to do with the similarities and differences between React’s constructor and the built in method getInitialState. While the simple answer to this question is indeed simple: “getInitialState is the ES5 friendly method to define the initial state of a React component,” there are a couple more details around getInitialState as well as React’s ES5 support that are interesting and useful to highlight.
One of the fundamental differences between ES5 and ES6 in regards to React implementation is the new class keyword. It allows definition of React components as classes, which is a familiar data structure for anyone who has had experience with more traditional object-oriented languages such as Java or C++. The class structure also allows for natural organization of the component’s elements like state, props, lifecycle methods and member functions. However, ES5 did not provide the same convenience. So instead of defining a React component as a class:
class HelloWorld extends React.Component {\n render() {\n return <span>Hello World</span>;\n }\n}You would rely on a helper module called create-react-class:
var createReactClass = require('create-react-class');\nvar HelloWorld = createReactClass({\n render: function() {\n return <span>Hello World</span>;\n }\n});And it is within the object passed into create-react-class that you could define an initial state by populating the getInitialState attribute:
var HelloWorld = createReactClass({\n getInitialState: function() {\n return {name: this.props.name};\n },\n render: function() {\n return <span>Hello {this.state.name}</span>;\n }\n});Which, in ES6 implementation would be the equivalent of:
\nclass HelloWorld extends React.Component {\n constructor(props) {\n super(props);\n this.state = {\n name: props.name\n }\n }\n render() {\n return <span>Hello {this.state.name}</span>;\n }\n}One difference worth noting is that the create-react-class method automatically binds this to every attribute method. This no longer holds true if you define React components using the common ES6 class syntax, making it so that you have to manually bind this to internal methods:
class HelloWorld extends React.Component {\n constructor(props) {\n super(props);\n this.state = {\n name: props.name\n }\n\n this.changeName = this.changeName.bind(this);\n }\n changeName() {\n this.setState({ name: "World" });\n }\n ...\n}Or otherwise use the “arrow function” shorthand which takes care of binding:
\nclass HelloWorld extends React.Component {\n constructor(props) {\n super(props);\n this.state = {\n name: props.name\n }\n }\n changeName = () => {\n this.setState({ name: "World" });\n }\n ...\n}Since the update to ES6, there have been multiple new React iterations. You could now forgo the constructor declaration altogether and just declare state inline as a class member, or utilize React Hooks as a new way to initialize states. However, the ES5 support remains useful for legacy systems and adds to the overall flexibility of React as a toolset.
You can read more about React's ES5 support in the official doc entry here, and the v0.13.0 beta release blog entry here, for the respective ES6 changes.
\n","frontmatter":{"date":"January 05, 2021","updated_date":null,"description":"No ES6? No problem. getInitialState is the ES5 friendly method to define the initial state of a React component.","title":"Constructor vs getInitialState in React","tags":["JavaScript","React"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/4fcf496b1fdd8d1d7ae98e1ab30f19b6/ee604/index.png","srcSet":"/static/4fcf496b1fdd8d1d7ae98e1ab30f19b6/69585/index.png 200w,\n/static/4fcf496b1fdd8d1d7ae98e1ab30f19b6/497c6/index.png 400w,\n/static/4fcf496b1fdd8d1d7ae98e1ab30f19b6/ee604/index.png 800w,\n/static/4fcf496b1fdd8d1d7ae98e1ab30f19b6/efc77/index.png 865w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Nathan Nguyen","github":"nathannguyenn","avatar":null}}}},{"node":{"excerpt":"In JavaScript, web workers allow developers to benefit from parallel programming. Parallel programming enables various computations to be…","fields":{"slug":"/engineering/web-workers/"},"html":"In JavaScript, web workers allow developers to benefit from parallel programming. Parallel programming enables various computations to be performed at the same time by applications.
\nWeb Workers are a browser feature that allows scripts to be executed on a separate thread from the main execution thread of your web application. This allows the main thread of your web application to run without being blocked by slow scripts in your application.
\nWell, sort of. This has been something that had confused me many a time when I was initially learning about JavaScript. JavaScript a synchronous, single-threaded language. However, JavaScript has features that allow you to execute asynchronous code, which is handled by browser engines (on your client) or by your OS (in NodeJS), which are capable of executing code in multiple threads.
\nWe work with asynchronous methods in JavaScript by either using callbacks, Promises or async/await. We'll use Promises as an example for exploring asynchronicity in JavaScript.
\nPromises are proxies for values that are not yet available when the Promise is created. This lets you organize parts of your code to run when the value becomes available or if something goes wrong.
\nThis does not mean that your code is running asynchronously. As I mentioned before, JavaScript code is executed on a single thread. The callback that processes the response from your created Promises still runs on your single main thread. Promises do, however, allow you to spawn asynchronous tasks, such as file I/O or making an HTTP request which runs outside of your code. This allows your main thread to work on something else while waiting for these tasks to return a response. This means that the callback functions which run after a response is received are called asynchronously.
\nI feel that the distinction between your code running asynchronously vs. being called asynchronously is important, as you will see that you are not able to perform computationally expensive tasks without blocking your main thread, even when using Promises.
\nHere is an example of using an expensive function (finding prime numbers) wrapped in a Promise.
\nfunction findPrimeNumbers(max) {\n return new Promise((resolve) => {\n console.log("starting findPrimeNumbers");\n let primes = [];\n for (let i = 0; i < max; i++) {\n if (i === 0 || i === 1) {\n continue;\n }\n let isPrime = true;\n for (let y = 2; y < i; ++y) {\n if (i % y === 0) {\n isPrime = false;\n break;\n }\n }\n if (isPrime) {\n primes.push(i);\n }\n }\n console.log("prime numbers found");\n resolve(primes);\n });\n}\n\nconsole.log("a");\nfindPrimeNumbers(100000).then(() => {\n console.log("b");\n});\nconsole.log("c");You'll find that the above program will print the following:
\na\nstarting findPrimeNumbers\nprime numbers found\nc\nbThe ordering of a, b, and c is not particularly noteworthy, as we already know that the callback for the findPrimeNumbers function will only execute after the main thread has nothing left to do, after printing c.
What I did find interesting was that the statement printing c waits until the findPrimeNumbers function completes before being executed. To prevent functions like this from blocking the main thread, we'll have to use Web Workers.
Web Workers run in a separate thread altogether, with no access to the context of the main thread. Data is sent between the worker and the main thread using a system of messages: the postMessage() method is used to send a message from one to the other, and the onmessage event handler is used to receive and respond to messages.
First we need to create the script (we'll just call it find-prime-numbers.js) that the Web Worker will run:
onmessage = (event) => {\n const { data } = event;\n const primes = findPrimeNumbers(parseInt(data));\n postMessage(primes);\n};\n\nfunction findPrimeNumbers(max) {\n console.log("starting findPrimeNumbers");\n let primes = [];\n for (let i = 0; i < max; i++) {\n if (i === 0 || i === 1) {\n continue;\n }\n let isPrime = true;\n for (let y = 2; y < i; ++y) {\n if (i % y === 0) {\n isPrime = false;\n break;\n }\n }\n if (isPrime) {\n primes.push(i);\n }\n }\n console.log("prime numbers found");\n return primes;\n} The onmessage handler receives an event with the message itself contained in the event's data attribute. Here we execute the findPrimeNumbers() function again, and use postMessage() function to send the result back to the main thread.
Over in your main script, we'll create a worker object using the constructor new Worker(), which runs the named JavaScript file containing the code that we just wrote, which will run in the worker thread.
const primeNumberWorker = new Worker("./find-prime-numbers.js");\nprimeNumberWorker.onmessage = (event) => {\n console.log("b");\n};\n\nconsole.log("a");\nprimeNumberWorker.postMessage("100000");\nconsole.log("c");Running this script will print the following:
\na\nc\nstarting findPrimeNumbers\nprime numbers found\nbYou can see that this time, the prime numbers function is actually running on a thread separate from our main one, which allows the main thread to continue printing the statement c while the worker thread works out its own response.
To reiterate, Web Workers allow you to run your own code on a separate thread, which allows your main thread to continue rendering the UI, receive user input, etc., while your worker thread processes a response.
\nFor most web applications, the most expensive operations in your code will often be I/O, interacting with resources in your network. Thanks to JavaScript's Event Loop, these are already offloaded from your main thread to your system's operating system. As such, there is no need for Web Workers when making these kinds of operations.
\nIf you need to process large sets of data, there is a decent chance that you will be able to perform this on your server-side before sending it to your front end client for display to the user. This will allow you to rely on JavaScript's event loop to prevent these expensive operations from blocking your main thread.
\nHowever, if your scenario must have these long-running tasks running on your client end, Web Workers would be an ideal solution for your needs. Some examples of these could be:
\nAs Web Workers work on a separate thread, it has no access to the parent page, which calls it. This means that the worker itself cannot manipulate the DOM nor access any data stored within it. It is not able to access the browser's localStorage and sessionStorage either. You are limited to the data sent directly via postMessage or IndexedDB for accessing stored client data.
As with anything multithreaded, adding Web Workers into your application will add complexity to your application. Natively, postMessage is the only direct line of communication between threads. You will need to define how more complex workflows will be managed between your main thread and worker thread (such as error handling).
If your worker can be triggered multiple times, each request will simply be queued and will run when the previous has been completed. For more complex queuing mechanisms, you would have to define this yourself.
\nWith discussions dating back a decade, Web Workers are not a new feature. Despite their powerful potential, they haven't been widely adopted. Client side applications, for the most part, just haven't needed to perform intensive tasks where Web Workers would have been a solution. In addition, with a lack of widespread use, finding relatable examples to reference when implementing them is an extra barrier for adoption.
\n","frontmatter":{"date":"December 28, 2020","updated_date":null,"description":"Learn how to use JavaScript web workers to create parallel programming and execute multiple operations concurrently rather than interconnecting them.","title":"Web Workers in JS - An Introductory Guide","tags":["Web Workers"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/a17120a4316c8564d40dee1e652c0641/ee604/unsplash.png","srcSet":"/static/a17120a4316c8564d40dee1e652c0641/69585/unsplash.png 200w,\n/static/a17120a4316c8564d40dee1e652c0641/497c6/unsplash.png 400w,\n/static/a17120a4316c8564d40dee1e652c0641/ee604/unsplash.png 800w,\n/static/a17120a4316c8564d40dee1e652c0641/f3583/unsplash.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Nick Chim","github":"nickc95","avatar":null}}}},{"node":{"excerpt":"Enum is a typed constant that is specified for the new data form of Enumeration. An effective way to describe a set of named integral…","fields":{"slug":"/engineering/enum-csharp/"},"html":"Enum is a typed constant that is specified for the new data form of Enumeration. An effective way to describe a set of named integral constants assigned to a variable is to include a Typesafe enumeration. Enums make the code more readable and less vulnerable to mistakes. If you have a set of functionally important and unchanged values, Enums are useful for developers.
\nEnums' key benefit is to make it possible in the future to adjust values. Enums are a robust alternative to the short String or Int constants used to describe sets of similar objects in far older APIs.
\nSometimes we must use the constant variable in our application to not be changed throughout the application. One of the methods to declare the continuous variables are Enum. Let's discuss it.
Enum is the short form of Enumeration. Enum consumes less memory and space because they are value types, which means they store memory value. Also, Enums are strongly typed named constants.
\nEnums written by developers are used in the .NET system to develop numeric constants. A numeric value must be allocated to all the members of an enum. Here are the main points about Enums:
\nEnums are of two types in C#
\nSimple Enum - The members of this enum contain a single value.
\nFlags Enum - The members of this enum contain multiple values or multiple values combined using a bitwise OR operator. These enums are often used for bitwise operators.
\nHere we will talk about the Simple Enum only.
\nEnum is declared by the enum keyword followed by the enum name and then its enum members. Like an example below
\npublic enum Vehicle\n{\n Car,\n Bike=3,\n Truck,\n Taxi,\n}In the above code, an enum for the Vehicle is created. The enum values are started from 0 by default if we do not assign them the values and incremented by 1 for the next enum member.
That means the Car enum has the value 0 and the Bike enum assigned the value 3 that means the Bike enum now has the value 3 and the next enums are incremented by 1 means Truck and Taxi enum have the value 4 and 5 respectively.
By default, the type for enum elements is int. We can set different type by adding a colon like an example below.
\npublic enum Vehicle: byte\n{\n Car,\n Bike=3,\n Truck,\n Taxi,\n}The different types which can be set are sbyte, byte, short, ushort, uint, ulong, and long.
\nTo get the value of enum we can simply typecast it to its type. In the first example, the default type is int so we have to typecast it to int. Also, we can get the string value of that enum by using the ToString() method as below.
Console.WriteLine((int)VehicleEnum.Car);\nstring vehicle = Vehicle.Bike.ToString();\nConsole.WriteLine(vehicle);Output:
\n0\nBikeFor parsing the string value into enum we have 2 methods
\nConsole.WriteLine(Enum.Parse(typeof(Vehicle),"Car"));Output:
\nCarout variable. If the string value is not parsed, then the result will be false.Enum.TryParse("0",out Vehicle value);\nConsole.WriteLine(value);Output:
\nCarNote - Both the methods also have overload methods.
\nWe can check if a given integral value, or its name as a string, exists in a specified enum.
\nConsole.WriteLine(Enum.IsDefined(typeof(Vehicle), 0));OR
\nConsole.WriteLine(Enum.IsDefined(typeof(Vehicle), "car"));Output:
\nTrueFor looping through the enums you can write the below code -
\nforeach (var data in Enum.GetNames(typeof(Vehicle)))\n{\n Console.WriteLine(data+" - "+ (int)Enum.Parse(typeof(Vehicle), data));\n}Here Enum.GetNames(Type enumType) method is used, which retrieves an array of the names from the specified enum.
\nOutput:
\nCar - 0\nBike - 3\nTruck - 4\nTaxi - 5We can call the enum member by its integral value. If there is no corresponding value related to that integral value, then it will print that integral value.
\n Console.WriteLine((Vehicle)3);Output
\nBikeThe advantages of using enums are that they are very easy to use and represented as strings but processed as integers. Enums are easy to maintain and improve code readability because they provide symbolic named constants, which means you need to remember the names, not the integer values.
\nIf you want to learn more about C programming, here is another article written on C# Exceptions and Exception Handling in C# I hope you learn something new today and will going to try it out, if you have any questions feel free to drop a comment below.
\n","frontmatter":{"date":"December 18, 2020","updated_date":null,"description":"Would you like to become more proficient in your C# programming in the use of enums? To learn the basics and use cases for Enum in C#, read this post.","title":"How to Use Enum in C#","tags":["C#","Enum"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/438545712e629112915b48eb3d068821/14b42/coverimage.jpg","srcSet":"/static/438545712e629112915b48eb3d068821/f836f/coverimage.jpg 200w,\n/static/438545712e629112915b48eb3d068821/2244e/coverimage.jpg 400w,\n/static/438545712e629112915b48eb3d068821/14b42/coverimage.jpg 800w,\n/static/438545712e629112915b48eb3d068821/9842e/coverimage.jpg 900w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Hemant Manwani","github":"hemant404","avatar":null}}}},{"node":{"excerpt":"The goal of this post is to learn about the various ways of data migration in MongoDB that can help us to write scripts that change your…","fields":{"slug":"/engineering/live-data-migration-mongodb/"},"html":"The goal of this post is to learn about the various ways of data migration in MongoDB that can help us to write scripts that change your database by adding new documents, modifying existing ones.
\nIf you're coming here for the first time, please take a look at the prequel Self-Hosted MongoDB.
\nAlright then, picking from where we left off, let's get started with the data migration in MongoDB.
\nNow, the basic steps to migrate data from one MongoDB to another would be:
\nThis is very straight forward when the source database is not online because we know that there won't be any new documents created/updated during the migration process.\nLet's look at simple migration first before diving into the live scenario.
\nWe're going to use an existing utility program mongodump for creating the database backup.
\nRun this command in the source database server
\nmongodump --host="hostname:port" \\\n --username="username" --password="password" \\\n --authenticationDatabase "admin" \\\n --db="db name" --collection="collection name" --query='json' \\\n --forceTableScan -v --gzip --out ./dump--host: The source MongoDB hostname along with the port. It defaults to localhost:27017. If it is a connection string you can use this option —-uri=\"mongodb://username:password@host1[:port1]...\"
--username: Specifies a username to authenticate to a MongoDB database that uses authentication.
--password: Specifies a password to authenticate to a MongoDB database that uses authentication.
--authenticationDatabase: Specifies the authentication database where the specified --username has been created.
\n\nIf you do not specify an authentication database or a database to export, mongodump assumes the admin database holds the user's credentials.
\n
--db: Specifies the database to take a backup from. If you do not specify a database, mongodump collects from all databases in this instance.
\n\nAlternatively, you can also specify the database directly in the URI connection string i.e.
\nmongodb://username:password@uri/dbname.
Providing a connection string while also using--dband specifying conflicting information will result in an error.
--collection: Specifies a collection to backup. If you do not specify a collection, this option copies all collections in the specified database or instance to the dump files.
--query : Provides a JSON document as a query that optionally limits the documents included in the output of mongodump.
\nYou must enclose the query document in single quotes ('{ ... }') to ensure that it does not interact with your environment.
\nThe query must be in Extended JSON v2 format (either relaxed or canonical/strict mode), including enclosing the field names and operators in quotes e.g. '{ \"created_at\": { \"\\$gte\": ISODate(...) } }'.
\n\nTo use the
\n--queryoption, you must also specify the--collectionoption.
--forceTableScan: Forces mongodump to scan the data store directly. Typically, mongodump saves entries as they appear in the index of the _id field.
\n\nIf you specify a query
\n--query, mongodump will use the most appropriate index to support that query.
Hence , you cannot use--forceTableScanwith the--queryoption.
--gzip: Compresses the output. If mongodump outputs to the dump directory, the new feature compresses the individual files. The files have the suffix .gz.
--out: Specifies the directory where mongodump will write BSON files for the dumped databases. By default, mongodump saves output files in a directory named dump in the current working directory.
We will use a utility program called mongorestore for restoring the database backup.
Copy the backup directory dump to the new Database instance and run the following command:
\nmongorestore --uri="mongodb://user:password@host:port/?authSource=admin" \\\n --drop --noIndexRestore --gzip -v ./dumpReplace the credentials with the new database credentials. Unline in the previous step, the --authenticationDatabase option is specified in the URI string.
Also, use --gzip if used while creating the backup.
--drop: Before restoring the collections from the dumped backup, drops the collections from the target database. It does not drop collections that are not in the backup.\n--noIndexRestore: Prevents mongorestore from restoring and building indexes as specified in the corresponding mongodump output.
\n\nIf you want to change name of the database while restoring, you can do so using
\n--nsFrom=\"old_name.*\" --nsTo=\"new_name.*\"options.
However, it won’t work if you were to migrate withoplogswhich is a requirement in migration from an online instance.
The only challenge with migrating from an online database is not able to pause the updates during migration. So here is the overview of the steps,
\noplogs capture\n\nNow, to capture
\noplogs, a replica set must be initialized in the source and destination databases. This is because theoplogsare captured fromlocal.oplog.rsnamespace, which is created after initializing a replica set.
You can follow this guide to configure a replica set.
Oplogs, in simple words, are the operation logs created per operation in the database. They represent a partial document state or, in other words, the database state. So we are going to capture any updates in our old database during the migration process using these oplogs.
Run the mongodump program with the following options,
\nmongodump --uri=".../?authSource=admin" \\\n --forceTableScan --oplog \\\n --gzip -v --out ./dump--oplog: Creates a file named oplog.bson as part of the mongodump output. The oplog.bson file, located in the top level of the output directory, contains oplog entries that occur during the mongodump operation. This file provides an effective point-in-time snapshot of the state of our database instance.
In order to replay the oplogs, a special role is required. Let's create and assign the role to the database user being used for migration.
\ndb.createRole({\n role: "interalUseOnlyOplogRestore",\n privileges: [\n {\n resource: { anyResource: true },\n actions: [ "anyAction" ] \n }\n ],\n roles: []\n})db.grantRolesToUser(\n "admin",\n [{ role:"interalUseOnlyOplogRestore", db:"admin" }]\n);Now you can restore using the mongorestore program with the following options,
\nmongorestore --uri="mongodb://admin:.../?authSource=admin" \\\n --oplogReplay \n --gzip -v ./dumpIn the above command, using the same user admin with whom the role was associated.
--oplogReplay: After restoring the database dump, replays the oplog entries from a bson file and restores the database to the point-in-time backup captured with the mongodump --oplog command.
Alright, so far we are done with most of the heavy lifting. The only thing that remains is maintaining consistency between the databases during the connection switch in our application servers.
\n\n\nIf you're running MongoDB version 3.6+, it's better to go for the Change Stream approach, which is a event-based mechanism introduced to capture changes in your database in an optimized way. Here is an article that covers it : An Introduction to Change Streams
\n
Check out the generic sync script, which you can run as a CRON job every minute.
\nUpdate the variables in this script and run as
\n$ ./delta-sync.sh from_epoch_in_milliseconds\n\n# from_epoch_in_milliseconds is automatically picked with every iteration if not suppliedOr you can set up a cron job to run this every minute.
\n* * * * * ~/delta-sync.shThe output can be monitored with the following command (I'm running RHEL 8, refer to your OS guide for cron output)
\n$ tail -f /var/log/cron | grep CRONThis is a sample sync log.
\nCMD (~/cron/dsync.sh)\nCMDOUT (INFO: Updated log registry to use new timestamp on next run.)\nCMDOUT (INFO: Created sync directory: /home/ec2-user/cron/dump/2020-11-03T19:01:01Z)\nCMDOUT (Fetching oplog in range [2020-11-03T19:00:01Z - 2020-11-03T19:01:01Z])\nCMDOUT (2020-11-03T19:01:02.319+0000#011dumping up to 1 collections in parallel)\nCMDOUT (2020-11-03T19:01:02.334+0000#011writing local.oplog.rs to /home/ec2-user/cron/dump/2020-11-03T19:01:01Z/local/oplog.rs.bson.gz)\nCMDOUT (2020-11-03T19:01:04.943+0000#011local.oplog.rs 0)\nCMDOUT (2020-11-03T19:01:04.964+0000#011local.oplog.rs 0)\nCMDOUT (2020-11-03T19:01:04.964+0000#011done dumping local.oplog.rs (0 documents))\nCMDOUT (INFO: Dump success!)\nCMDOUT (INFO: Replaying oplogs...)\nCMDOUT (2020-11-03T19:01:05.030+0000#011using write concern: &{majority false 0})\nCMDOUT (2020-11-03T19:01:05.054+0000#011will listen for SIGTERM, SIGINT, and SIGKILL)\nCMDOUT (2020-11-03T19:01:05.055+0000#011connected to node type: standalone)\nCMDOUT (2020-11-03T19:01:05.055+0000#011mongorestore target is a directory, not a file)\nCMDOUT (2020-11-03T19:01:05.055+0000#011preparing collections to restore from)\nCMDOUT (2020-11-03T19:01:05.055+0000#011found collection local.oplog.rs bson to restore to local.oplog.rs)\nCMDOUT (2020-11-03T19:01:05.055+0000#011found collection metadata from local.oplog.rs to restore to local.oplog.rs)\nCMDOUT (2020-11-03T19:01:05.055+0000#011restoring up to 4 collections in parallel)\nCMDOUT (2020-11-03T19:01:05.055+0000#011replaying oplog)\nCMDOUT (2020-11-03T19:01:05.055+0000#011applied 0 oplog entries)\nCMDOUT (2020-11-03T19:01:05.055+0000#0110 document(s) restored successfully. 0 document(s) failed to restore.)\nCMDOUT (INFO: Restore success!)You can stop this script after verifying that no more oplogs are being created, i.e., when source DB went offline.
This concludes the complete self-hosted MongoDB data migration guide. If you want to learn more about MongoDB here is a useful resource on how to use MongoDB as datasource in goLang.
\n","frontmatter":{"date":"December 14, 2020","updated_date":null,"description":"This article covers the guide to migrate data from offline or live MongoDB instance using oplog replay alongside mitigate connection switch latency with existing utilities.","title":"How to Migrate Data In MongoDB","tags":["MongoDB"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.4184397163120568,"src":"/static/90f92104129cbc152fa1d9a64c58d7be/ee604/index.png","srcSet":"/static/90f92104129cbc152fa1d9a64c58d7be/69585/index.png 200w,\n/static/90f92104129cbc152fa1d9a64c58d7be/497c6/index.png 400w,\n/static/90f92104129cbc152fa1d9a64c58d7be/ee604/index.png 800w,\n/static/90f92104129cbc152fa1d9a64c58d7be/31987/index.png 1000w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Chinmaya Pati","github":"cnp96","avatar":null}}}},{"node":{"excerpt":"This guide uses LoginRadius API for authenticating React apps. It provides React developers with a more straightforward way to add user…","fields":{"slug":"/engineering/user-authentication-react-application/"},"html":"This guide uses LoginRadius API for authenticating React apps. It provides React developers with a more straightforward way to add user authentication to react apps. To handle a lot of authentication implementation information, LoginRadius offers a high-level API. Using security best practices, now you can protect your response apps while writing less code.
\nThis article focuses on helping developers learn how to integrate user authentication in the React application. Practice the following security principles to improve authentication on React applications:
\nIn the React application, we can easily and quickly add authentication by using LoginRadius. If you already have a customized login/registration page ready, then I'll guide you to the next step of adding authentication in react apps.
\nA new application was created for you when you signed up for LoginRadius. From here, you get some essential information.
\n<script type="text/javascript" src="https://auth.lrcontent.com/v2/LoginRadiusV2.js"></script>We will use an API framework to call LoginRadius APIs for authentication. Create a new file LoginPage.js, and add the following code.
\n import React, { useState } from "react";\n const lrconfig = {\n apiKey: "*************************", //LoginRadius API key\n };\n const loginradius = {};\n if (window.LoginRadiusV2) {\n loginradius = new window.LoginRadiusV2(lrconfig);\n loginradius.api.init(lrconfig);\n }\n const LoginButton = () => {\n const loginButtonHandler = () => {\n loginradius.api.login({emailid: emailValue,\n password: passwordValue},\n (successResponse)=>{\n //Here you will get the access Token of \n \n \n console.log(successResponse);\n },\n (errors) => {\n console.log(errors);\n });\n }\n const [emailValue, updateEmailValue] = useState("");\n const [passwordValue, updatePasswordValue] = useState("");\n return (\n <React.Fragment>\n <input type="text" value={emailValue} onChange={(e)=>{updateEmailValue(e.target.value)}} placeholder={"email"}/>\n <input type="password" value={passwordValue} onChange={(e)=>{updatePasswordValue(e.target.value)}} placeholder={"Password"} />\n <button onClick={() => loginButtonHandler()}>Log In</button>\n </React.Fragment>\n );\n };\n export default LoginButton;In the above code, you will see the lrConfig object, which has apikey that you will get from the LoginRadius account. After calling loginradius.api.login, you will get the response in which you will get the access Token. Through this access token, you can get the user profile.
Create LogoutPage.js file and add following code:
\n import React, { useState } from "react";\n\n const lrconfig = {\n apiKey: "*************************", //LoginRadius API key\n };\n\n const loginradius = {};\n if (window.LoginRadiusV2) {\n loginradius = new window.LoginRadiusV2(lrconfig);\n loginradius.api.init(lrconfig);\n }\n\n const LogoutButton = () => {\n const token = '************'; // Access Token that you got after login\n const logoutButtonHandler = () => {\n //Note: Call invalidate token api to invalidate the token.**\n loginradius.api.invalidateToken(\n token,\n (successResponse)=>{\n console.log(successResponse);\n },\n (errors) => {\n console.log(errors);\n }\n );\n }\n return (\n <React.Fragment>\n <button onClick={() => logoutButtonHandler()}>Logout</button>\n </React.Fragment>\n );\n };\n export default LogoutButton;In the above code, we have called invalidated token api, which expires your access token.
\nCreate SignupPage.js file and add the following code:
\n import React, { useState } from "react";\n const lrconfig = {\n apiKey: "*************************", //LoginRadius API key\n sott: "***************************" //Secure Token for signup functionality\n };\n const loginradius = {};\n if (window.LoginRadiusV2) {\n loginradius = new window.LoginRadiusV2(lrconfig);\n loginradius.api.init(lrconfig);\n }\n const SignupButton = () => {\n const signupButtonHandler = () => {\n loginradius.api.registration({\n email: [{ type: "Primary", value: emailValue }],\n password: passwordValue\n },\n (successResponse)=>{\n //Here you will get the response after registration\n console.log(successResponse);\n },\n (errors) => {\n console.log(errors);\n });\n }\n const [emailValue, updateEmailValue] = useState("");\n const [passwordValue, updatePasswordValue] = useState("");\n return (\n <React.Fragment>\n <input type="text" value={emailValue} onChange={(e)=>{updateEmailValue(e.target.value)}} placeholder={"email"}/>\n <input type="password" value={passwordValue} onChange={(e)=>{updatePasswordValue(e.target.value)}} placeholder={"Password"} />\n <button onClick={() => signupButtonHandler()}>Log In</button>\n </React.Fragment>\n );\n };\n export default SignupButton;In the above code, You will get a success/error response after calling the registration api.
\nThe most common authentication use case for a React application gets covered in this tutorial: quick login and logout. However, LoginRadius is an expandable and versatile platform that can help you accomplish much more. In this guide, We have used the LoginRadius API Framework. If you want to incorporate our hosted page into your application, follow this documentation.
\nLet me know what you think of this tutorial in the comments below. Thanks for reading :)
\n","frontmatter":{"date":"December 10, 2020","updated_date":null,"description":"This article focuses on helping developers learn how to integrate user authentication in React applications and determine the basic principles of authentication with React.","title":"A Guide To React User Authentication with LoginRadius","tags":["Authentication","LoginRadius","React"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/1d36656dce223836227b1b52270427a8/ee604/authentication-main.png","srcSet":"/static/1d36656dce223836227b1b52270427a8/69585/authentication-main.png 200w,\n/static/1d36656dce223836227b1b52270427a8/497c6/authentication-main.png 400w,\n/static/1d36656dce223836227b1b52270427a8/ee604/authentication-main.png 800w,\n/static/1d36656dce223836227b1b52270427a8/f3583/authentication-main.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Versha Gupta","github":"vershagupta","avatar":null}}}}]},"markdownRemark":{"excerpt":"Google has prepared a roadmap to restrict third-party cookies in Chrome. Since 04 January 2024, Chrome has rolled out third-party cookie…","fields":{"slug":"/engineering/identity-impact-of-google-chrome-thirdparty-cookie-restrictions/"},"html":"Google has prepared a roadmap to restrict third-party cookies in Chrome. Since 04 January 2024, Chrome has rolled out third-party cookie restrictions for 1% of stable clients and 20% of Canary, Dev, and Beta clients.
\nWhat does it mean for user authentication?
\nOn one hand, Google believes third-party cookies are widely used for cross-site tracking, greatly affecting user privacy. Hence, Google wants to phase out (or restrict) supporting third-party cookies in Chrome by early Q2 2025 (subject to regulatory processes).
\nOn the other hand, Google introduced Privacy Sandbox to support the use cases (other than cross-site tracking and advertising) previously implemented using third-party cookies.
\nIn this article, we’ll discuss:
\nThird-party cookie restrictions affect user authentication in three ways, as follows.
\nIf your website or app uses an external Identity Provider (IdP) — like LoginRadius, the IdP sets a third-party cookie when the user authenticates on your app.
\nIf you have multiple apps across domains within your organization and authentication is handled using an IdP (internal or external) with web SSO, you already use third-party cookies to facilitate seamless access for each user using a single set of credentials.
\nIf you have implemented web SSO with one primary domain and multiple sub-domains of the primary domain, third-party cookie restrictions may not apply. For now, Google doesn’t consider the cookies set by sub-domains as third-party cookies, although this stance may change in the future.
\nFor example, you have apps at example.com, travel.example.com, stay.example.com, and web SSO is handled by auth.example.com. In this case, third-party cookie restrictions don’t apply.
Federated SSO is similar to, albeit different from, web SSO. It can handle multiple IdPs and applications—aka., Service Providers (SPs)—spanning multiple organizations. It can also implement authentication scenarios that are usually implemented through web SSO.
\nUsually, authentication is handled on a separate pop-up or page when the user wants to authenticate rather than on the application or website a user visits.
\nFor example, you already use federated SSO if you facilitate authentication for a set of apps through multiple social identity providers as well as traditional usernames and passwords.
\n\n\nNote: It is also possible to store tokens locally, not within cookies. In this case, third-party cookie restrictions won’t affect token-based authentication. However, the restrictions still affect authentication where tokens are stored within third-party cookies (a common and secure method).
\n
Google has been developing alternative features and capabilities for Chrome to replace third-party cookies as part of its Privacy Sandbox for Web initiative.
\nSpecific to authentication, Google recommends the following:
\nCHIPS are a restricted way of setting third-party cookies on a top-level site without making them accessible on other top-level sites. Thus, they limit cross-site tracking and enable specific cross-site functionalities, such as maps, chat, and payment embeds.
\nFor example, a user visits a.com with a map embed from map-example.com, which can set a partitioned cookie that is only accessible on a.com.
If the user visits b.com with a map embed from map-example.com, it cannot access the partitioned cookie set on a.com. It has to create a separate partitioned cookie specific to b.com, thus blocking cross-site tracking yet allowing limited cross-site functionality.
You should specifically opt for partitioned cookies (CHIPS), which are set with partitioned and secure cookie attributes.
\nIf you’re using an external identity provider for your application, CHIPS is a good option to supplant third-party cookie restrictions.
\nHowever, CHIPS may not be ideal if you have a web SSO or federated SSO implementation. It creates separate partitioned cookies for each application with a separate domain, which can increase complexity and create compatibility issues.
\nWith Storage Access API, you can access the local storage in a third-party context through iframes, similar to when users visit it as a top-level site in a first-party context. That is, it gives access to unpartitioned cookies and storage.
\nStorage Access API requires explicit user approval to grant access, similar to locations, camera, and microphone permissions. If the user denies access, unpartitioned cookies and storage won’t be accessible in a third-party context.
\nIt is most suitable when loading cross-site resources and interactions, such as:
\nVerifying user sessions when allowing interactions on an embedded social post or providing personalization for an embedded video.\nEmbedded documents requiring user verification status to be accessible.
\nAs it requires explicit user approval, it is advisable to use Storage Access API when you can’t implement an identity use case with the other options.
\nWith Related Website Sets, you can declare a primary website and associatedSites for limited purposes to grant third-party cookie access and local storage for a limited number of sites.
Chrome automatically recognizes related website sets declared, accepted, and maintained in this open-source GitHub repository: Related Website Sets
\nIt provides access through Storage Access API directly without prompting for user approval, but only after the user interacts with the relevant iframe.
\nIt is important to declare a limited number of domains in related website sets that are meaningful and used for specific purposes. Google may block or suspend any exploitative use of this feature.
\nThe top-level site can also request approval for specific cross-site resources and scripts to Storage Access API using resuestStorageAccessFor() API.
If you’re using an external identity provider for your web application, you can declare the domain of the identity provider in the related set to ensure limited third-party cookies and storage access to the identity provider, thus ensuring seamless user authentication.
\nRelated Website Sets can also work to supplement third-party cookie restrictions in web SSO and federated SSO if the number of web applications (or domains) is limited.
\nFedCM API enables federated SSO without third-party cookies.
\nWith FedCM API, a user follows these steps for authentication:
\nYou can access a user demo of FedCM here: FedCM.
\nFor more information about implementing federated SSO with FedCM API, go through the FedCM developer guide.
\nFirstly, we’re committed to solving our customers' user identity pain points — and preparing for the third-party cookies phase-out is no different.
\nWe’ll implement the most relevant and widely useful solutions to facilitate a smooth transition for our customers.
\nPlease subscribe to our blog for more information. We’ll update you on how we help with the third-party cookie phase-out.
\nThe proposed changes to phase out third-party cookies and suggested alternatives are evolving as Google has been actively collaborating and discussing changes with the border community.
\nMoreover, browsers like Firefox, Safari, and Edge may approach restricting third-party cookies differently than Google does.
\nFrom LoginRadius, we’ll keep you updated on what we’re doing as a leading Customer Identity and Access Management (CIAM) vendor to prepare for the third-party cookie phase-out.
\nTop-level site: It is the primary site a user has visited.
\nFirst-party cookie: A cookie set by the top-level site.
\nThird-party cookie: A cookie set by a domain other than the top-level site. For example, let’s assume that a user has visited a.com, which might use an embed from loginradius.com to facilitate authentication. If loginradius.com sets a cookie when the user visits a.com, it is called a third-party cookie as the user hasn’t directly visited loginradius.com.