Before we get into details of finding out optimal clusters, let's first see what the KMeans clustering algorithm is and some basics about it.
\nClustering is an unsupervised ML technique wherein we cluster the data to get insights from it. Clustering the data is quite essential for some business models and problems. It gives us conclusions on what is a cluster, i.e. data which is similar and in the form of cluster or groups.
\n\n\nClustering is the process of dividing the entire data into groups (also known as clusters) based on the patterns in the data.
\n
It is an algorithm for clustering. We will be discussing this method with code in the further sections.
\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.cluster import KMeans\n%matplotlib inlineNow let's discuss the method behind finding out the right number of clusters on a K-Means clustering algorithm.\nSo we will learn how to decide what number of clusters to input into your K-Means algorithm.\nHere we've got a data science problem.\nWe've got only two variables, x and y coordinates.
\nNow, if we run the K means clustering algorithm on this dataset with three clusters or with K pre-determine the clusters to be three, then the result will look something like this.
\nSo you can see here that it jumps from 8000 down to 3000, that's a massive change of 5000. Let's just call them units, 5000 units and then from 3000 as we increase the number of the close from 2 to 3, they jump from 3000 to 1000.
\nAgain quite a large drop and then from three to four what's going to happen is going to jump from 1000 to maybe eight hundred and from 800 to 600, 600 to 500 and so on so as you can see the first two improvements or first two changes from one cluster to two from two to three created some huge jumps or considerable drops in the WTS going forward The WCR says drops not substantially. And this is our hint at selecting the optimum optimal number of clusters; and the method we're going to use is the elbow method, and it is very visual. All you have to do is look at your chart and look for that change, or that's kind of like it does look like an ELBOW.
\nLook for that elbow in your chart where the drop goes from being quite substantial to being not as significant not as proven is not as great, and therefore, that point in your chart will be the optimal number of clusters.
\nIn this case, it is indeed three clusters.
\nThat is the optimal number. And as you can imagine, this method is entirely arbitrary.\nSometimes, the situations are not as pronounced as the elbow might not be as evident as in this case, and therefore, somebody might pick one number of clusters. Someone else might come along and select a different number.
\nfrom sklearn.cluster import KMeans\nwcss = []\n\nfor i in range(1, 11):\n\n kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 42)\n\n kmeans.fit(X)\n \n wcss.append(kmeans.inertia_)\n\nplt.plot(range(1, 11), wcss)\n\nplt.title('The Elbow Method')\n\nplt.xlabel('Number of clusters')\n\nplt.ylabel('WCSS')\n\nplt.show()When building APIs, the need to upload files is expected, which can be images, text documents, scripts, pdfs, among others. In the development of this functionality, some problems can be found, such as the number of files, valid file types, sizes of these files, and several others. And to save us from these problems we have the Multer library. Multer is a node.js middleware for handling multipart/form-data that is used to send files in forms.
The first step is to create a NodeJS project on your computer.
\nIn your terminal, type the following command:
\nyarn add express* You can also use NPM for installation
\nCreate a file named app.js inside the src/ folder. The next step is to start our Express server in our app.js
const express = require("express")\nconst app = express()\n\napp.listen(3000 || process.env.PORT, () => {\n console.log("Server on...")\n})With the project created, configured and with Express installed, we will add the multer to our project.
\nyarn add multerThe next step is to import the multer into our app.js file.
const multer = require("multer")We are almost there. Now create a folder called uploads where we will store the uploaded files.
Now we are at a very important stage which is the configuration of diskStorage. DiskStorage is a method made available by multer where we configure the destination of the file, the name of the file and we can also add validations regarding the type of the file. These settings are according to the needs of your project. Below I will leave an elementary example of the configuration.
const storage = multer.diskStorage({\n destination: (req, file, cb) => {\n cb(null, "uploads/")\n },\n filename: (req, file, cb) => {\n cb(null, Date.now() + "-" + file.originalname)\n },\n})In the configuration above, we mentioned the destination for the uploaded files and also change the name of the file .
\nconst uploadStorage = multer({ storage: storage })\n\n// Single file\napp.post("/upload/single", uploadStorage.single("file"), (req, res) => {\n console.log(req.file)\n return res.send("Single file")\n})\n\n//Multiple files\napp.post("/upload/multiple", uploadStorage.array("file", 10), (req, res) => {\n console.log(req.files)\n return res.send("Multiple files")\n})In the code snippet above, we created 2 POST routes for sending files. The first /upload/single route receives only a single file, note that the uploadStorage variable receives our diskStorage settings. As a middleware in the route, it calls the single method for uploading a single file. The /upload/multiple route receives several files, but with a maximum limit of 10 files, note that the uploadStorage variable now calls the ʻarray` method for uploading multiple files.
With all the settings done, our little API is already able to store the files sent.
\nconst express = require("express")\nconst multer = require("multer")\n\nconst app = express()\n\nconst storage = multer.diskStorage({\n destination: (req, file, cb) => {\n cb(null, "uploads/")\n },\n filename: (req, file, cb) => {\n cb(null, Date.now() + "-" + file.originalname)\n },\n})\n\nconst uploadStorage = multer({ storage: storage })\n\n// Single file\napp.post("/upload/single", uploadStorage.single("file"), (req, res) => {\n console.log(req.file)\n return res.send("Single file")\n})\n//Multiple files\napp.post("/upload/multiple", uploadStorage.array("file", 10), (req, res) => {\n console.log(req.files)\n return res.send("Multiple files")\n})\n\napp.listen(3000 || process.env.PORT, () => {\n console.log("Server on...")\n})Now it's up to you!
\n","frontmatter":{"date":"October 12, 2020","updated_date":null,"description":"Learn how to upload files in a NodeJS application using Multer, Multer is a middleware for handling multipart/form-data that is used to send files in forms.","title":"Upload files using NodeJS + Multer","tags":["NodeJs","Express","Multer"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/49a3115a8c11e7fd9aca612e846c5936/ee604/node-multer-upload.png","srcSet":"/static/49a3115a8c11e7fd9aca612e846c5936/69585/node-multer-upload.png 200w,\n/static/49a3115a8c11e7fd9aca612e846c5936/497c6/node-multer-upload.png 400w,\n/static/49a3115a8c11e7fd9aca612e846c5936/ee604/node-multer-upload.png 800w,\n/static/49a3115a8c11e7fd9aca612e846c5936/f3583/node-multer-upload.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Gabriel Rabelo","github":"gabrielrab","avatar":null}}}},{"node":{"excerpt":"Introduction Learning Deep Features for Discriminative Localization: Machine learning and Deep learning are gaining traction in today’s…","fields":{"slug":"/engineering/class-activation-mapping/"},"html":"Machine learning and Deep learning are gaining traction in today’s world and are making significant and unimaginable progress in almost every industry. However, with the increase in complexity and accuracy of these algorithms, the interpretability of these is at stake- especially the deep learning models which take in more than a million parameters for complex, convoluted models. Class Activation Mapping (CAM) is one such technique which helps us in enhancing the interpretability of such complex models.
\nFor a particular class (or category), Class activation mapping basically indicates the discriminative region of the image, which influenced the deep learning model to make the decision. The architecture is very similar to a convolutional neural network. It comprises several convolution layers, with the layer just before the final output performing Global Average Pooling. The features that are obtained are fed into the fully connected neural network layer governed by the softmax activation function and thus, output us the required probabilities. The importance of the weights with respect to a category can be found out by projecting back the weights onto the last convolution layer’s feature map.
\nThe Global Average Pooling (GAP) is preferred over Global Max Pooling (GMP) because GAP helps us in identifying the full extent of the object as compared to the GMP layer, which identifies one discriminative part. In Global Average Pooling, an average is taken across all the activation maps which help us to find all the possible discriminative regions present in them. Contrary to this, the Global Max Pooling method just considers the most discriminative region. Hence, Global Average Pooling showed better results than Global Max Pooling.
\nLet be the activation map of unit
in the last convolutional layer at spatial location
.
The result of GAP is represented as:-
\nFor a class c, an input to the softmax will be:-
\nOutput of Softmax layer:-
\nThus, the final equation for an activation map of class c would be:-
\n
The localization ability of the CAM method was put to the test when they were trained on the ILSVRC 2014 benchmark dataset. The CAM technique was used on popular CNN models like AlexNet, VGGNet and GoogLeNet by tweaking their models and fitting a GAP layer (similar to the CAM architecture) towards the end. This modified model was giving astounding results with the GAP layer as compared to their traditional architecture in terms of discriminative localization.
\nAfter applying a CAM architecture to fine-grained recognition and pattern discovery (like discovering informative objects in the scenes, concept localization in weakly labelled images, weakly supervised text detector and interpreting visual question answering), we can infer that feature capturing and localization was far more accurate in the CAM based GAP layer architecture, as the complete extent of the features were captured.\nVisualizing Class-specific Units:-\nWhen we use the GAP layer and the ranked softmax weight, we can directly visualize the units, which are the most discriminative for a particular class. Thus, CNN actually learns a bag of words, where each word is a discriminative class-specific unit. A combination of these class-specific units helps to guide CNNs in classifying each image.
\nCAMs are a great technique to interpret the information from the CNN models. However, the disadvantage of CAMs is that they can be noisy, and there might be some loss of spatial information. Hence, the Grad-CAM architecture and the Score-CAM architecture were built upon the CAM architecture to improve the accuracy, feature capturing and retain precise spatial information.
\n","frontmatter":{"date":"October 10, 2020","updated_date":null,"description":"Learn about the importance of the explainability of deep learning models and Class Activation Map Technique","title":"Class Activation Mapping in Deep Learning","tags":["Explainable AI","Deep Learning","CNN","Machine Learning"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/899d533ed44a50ce8e5dbf8103a0d717/ee604/Cover.png","srcSet":"/static/899d533ed44a50ce8e5dbf8103a0d717/69585/Cover.png 200w,\n/static/899d533ed44a50ce8e5dbf8103a0d717/497c6/Cover.png 400w,\n/static/899d533ed44a50ce8e5dbf8103a0d717/ee604/Cover.png 800w,\n/static/899d533ed44a50ce8e5dbf8103a0d717/f3583/Cover.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Ankit Choraria","github":"Ankit810","avatar":null}}}},{"node":{"excerpt":"What is data enrichment? and its importance Data enrichment is the process of combining first-party data from internal sources with…","fields":{"slug":"/engineering/full-data-science-pipeline-implementation/"},"html":"Data enrichment is the process of combining first-party data from internal sources with disparate data from other internal systems or third-party data from external sources.
\nUsually, the data available from clients or stakeholders are not enough to solve the given problem statement, like if a client comes with a problem statement to build a recommendation engine for his mutual fund industry, the usual data they have is old purchase data but that's not enough as client behaviour changes with time and is impacted by the present market condition, oil prices, etc. which needs to be incorporated in the model to make it efficient.
\nCodes for this tutorial is at Link
\nThe whole process id divided into four steps:
\nI have implemented a full pipeline of data science from scrapping data from web to implementing ml and NLP classification.
\nHere I have scraped data from IMDB website (imdb.py)
\nI have tried to implement simple ML regression on the data (ml_imdb.py)
\nI have prepared the data for NLP classification (multilabel_prep.py)
\nI have implemented multilabel NLP classifier using various techniques like chain classifier etc. (multilabelnlpclassifier.ipynb)
\nWeb scraping is the process of extracting and parsing raw data from the web. Web scraping is a technique which helps data scientist to make their data-rich and is an efficient technique of data collection.
\nThis world is full of data, but unfortunately, most of them are not in the form to be used. Data is like crude oil, or we say it is in unstructured form. For a data scientist or engineer, our first challenge is to make the data model consumption ready, which takes the majority of the time, and this whole process is collectively known as data preprocessing.
\nHTML is a form of primary markup language and the base framework of mostly all websites. For performing web scraping its necessary to know it
\nHere we will start with requesting the web page using python package requests.
\n from requests import get\n url = 'http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1'\n response = get(url)\n print(len(response.text))The whole web page is now stored in the variable object response.\nThen we parse the web page using beautifulsoup package.
\n from bs4 import BeautifulSoup\n html_soup = BeautifulSoup(response.text, 'html.parser')\n type(html_soup)Then I will store all the div with the class named lister-item mode-advanced in variable movie_containers.
\nmovie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')Then I iterate through this object and store the information in lists to make my final DataFrame, using simple for loops.
\n# Lists to store the scraped data in\nnames = []\nyears = []\nimdb_ratings = []\nmetascores = []\nvotes = []\n#gross=[] #many movies have no record\nmovie_description=[]\nmovie_duration=[]\nmovie_genre=[]\n# Extract data from individual movie container\nfor container in movie_containers:\n# If the movie has Metascore, then extract:\n if container.find('div', class_ = 'ratings-metascore') is not None:\n# The name\n name = container.h3.a.text\n names.append(name)\n# The year\n year = container.h3.find('span', class_ = 'lister-item-year').text\n years.append(year)\n# The IMDB rating\n imdb = float(container.strong.text)\n imdb_ratings.append(imdb)\n# The Metascore\n m_score = container.find('span', class_ = 'metascore').text\n metascores.append(int(m_score))\n# The number of votes\n vote = container.find('span', attrs = {'name':'nv'})['data-value']\n votes.append(int(vote))\n# Gross income of movie\n #gross_inc =container.find_all('span', attrs = {'name':'nv'})[1]['data-value']\n #gross.append(gross_inc)\n\n# movie description\n movie_desc=container.find_all('p', class_ = 'text-muted')[1].text\n movie_description.append(movie_desc)\n movie_det=container.find_all('p', class_ = 'text-muted')[0]\n\n\n# Movie duration\n movie_dur=movie_det.find('span',class_='runtime').text\n movie_duration.append(movie_dur)\n\n# Movie genre\n movie_gen=movie_det.find('span',class_='genre').text\n movie_genre.append(movie_gen)\n\nimport pandas as pd\none_df = pd.DataFrame({'movie': names,\n'year': years,\n'imdb': imdb_ratings,\n'metascore': metascores,\n'votes': votes,\n#'gross':gross,\n'movie decription':movie_description,\n'movie duration':movie_duration,\n'movie genre':movie_genre\n})\nprint(one_df.info())\none_df.to_csv('50_movie_details.csv')But this was only for one page which has data for 50 movies only which is not enough to build a model.
\nPlease refer my code to understand how I use simple for loops to iterate through all the movies and downloading data for 20 years(approx).
\nWhats is linear regression??
\nIt is one of the most popular and used statistical techniques\n• Used to understand the relationship between variables
\n• Can also be used to predict a value of interest for new observations
\n• The aim is to predict the value of a continuous numeric variable of interest (known as the response or dependent or target variable)
\n• The values of one or more predictor (or independent) variables are used to make the prediction
\n• One predictor = simple regression
\n• More predictors = multiple regression
\nHere I just tried to use metascore of movies firstly to predict IMDB ratings and secondly I wanted to enhance it by using metascore and votes to predict IMDB rating.
\n## ML model\n\nX = data.loc[:, 'metascore'].values\ny = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train.reshape(-1,1), y_train.reshape(-1,1))#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test.reshape(-1,1))\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n\n# 0.18041462828221905## Let try with imdb and votes\nX1 = data.loc[:, ['metascore','votes']].values\ny1 = data.loc[:, 'imdb'].values\n\n# Splitting the dataset into the Training set and Test set\nfrom sklearn.cross_validation import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X1, y1, test_size = 0.33, random_state = 0)\n\nfrom sklearn.linear_model import LinearRegression\n\nregressor = LinearRegression()#making object for reg package\nregressor.fit(X_train, y_train)#to fit the regressor to our training data\n\n#predict the test results\ny_pred =regressor.predict(X_test)\n\n\nfrom sklearn.metrics import mean_squared_error\nmean_squared_error(y_test, y_pred)\n# 0.15729132122310804 good scoreI tried to scrape data from the IMDB site and then applied ML regression techniques on it. Later I found that the movies listed are multi-class like Logan belongs to Action, Drama, Sci-Fi, which led me to think about how to implement the classifier model in the multilabel data. Usually, the data we get in real-world is mostly multi labelled like chatbot data; the intent is many and like these movies which are multi-class.
\nHere we will first see how we prep our data for multilabel classification.
\nHere we have all tags in one single column which is not usable while we do classification, so we have to make separate columns for all labels, and if the row doesn't belong to that category, it will be filled by 0 else 1.
\nimport os\nos.chdir('Desktop/web_scraping/imdb scrapper_ml/')\n\nimport pandas as pd\n\ndata=pd.read_csv('multilabel_nlp_classification.csv')\n\nmovie_list=[x for x in data['movie genre']]\nmovie_list1=''\nfor x in data['movie genre']:\n movie_list1+=','+x\n\nli_m=movie_list1.split(',')\nli=[x.strip() for x in li_m]\nlist_s=list(set(li))\n\nfor x in list_s:\n data[x]=0\n\ndata['movie_genre']=[x.strip().split(',') for x in data['movie genre']]\n\nde=data.copy()\n#data.loc[0,'Action']=1\nde['id']=range(0,6116)\n#print(de.loc[de['id']==0,'Action'])\nfor i in range(0,6116):\n for x in de.loc[de['id']==i,'movie_genre']:\n for y in x:\n y=y.strip()\n de.loc[de['id']==i,y]=1\n\nde.to_csv('multilabel_nlp_classification.csv')Now, as our data is ready, we can start with NLP implementation.
\nFor multilabel classification, I used techniques like classifier chain, label powerset, etc.
\nHere the problem statement is that using the movie description our model has to guess which genre the movie belongs to. It is a popular use case. Take an example of ecommerce product description data; now instead of manually assigning the labels to it, we can use a model which will find relevant labels or genre for it and make the content relevant to the type it belongs.
\nI start with Exploratory data analysis and then data cleaning, which is the most crucial step as if all the description has some very 30-50 common words it will simply make the data-heavy and model slow and inefficient.
\nThen we go on to make the data model ready as ML models don't understand text data we have to feed numbers in it. For that purpose, we use TfidfVectorizer.
\nTfidfVectorizer - Transforms text to feature vectors that can be used as input to the estimator.
\nThen simply diving the data in train and test split.
\nx_train = vectorizer.transform(train_text)\ny_train = train.drop(labels = ['id','movie decription'], axis=1)\n\nx_test = vectorizer.transform(test_text)\ny_test = test.drop(labels = ['id','movie decription'], axis=1)I tried first with applying logistic regression and one vs rest classifier.
\nOneVsRestClassifier strategy splits a multi-class classification into one binary classification problem per class.\nOneVsRestClassifier is when we want to do multi-class or multilabel classification, and its strategy consists of fitting one classifier per class. For each classifier, the class is fitted against all the other classes.
\n# Using pipeline for applying logistic regression and one vs rest classifier\nLogReg_pipeline = Pipeline([\n ('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),\n ])\n\nfor category in categories:\n printmd('**Processing {} comments...**'.format(category))\n \n # Training logistic regression model on train data\n LogReg_pipeline.fit(x_train, train[category])\n \n # calculating test accuracy\n prediction = LogReg_pipeline.predict(x_test)\n print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))\n print("\\n")Next, I tried with BinaryRelevance
\nIt is a simple technique which treats each label as a separate single class classification problem.
\n# using binary relevance\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom sklearn.naive_bayes import GaussianNB\n\n# initialize binary relevance multi-label classifier\n# with a gaussian naive bayes base classifier\nclassifier = BinaryRelevance(GaussianNB())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using ClassifierChain.
\nIt is almost similar to BinaryRelevance, here the first classifier is trained just on the input data, and then each next classifier is trained on the input space and all the previous classifiers in the chain.
\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom sklearn.linear_model import LogisticRegression\n\n\n# initialize classifier chains multi-label classifier\nclassifier = ClassifierChain(LogisticRegression())\n\n# Training logistic regression model on train data\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Next, I tried using Label Powerset.
\nHere we transform the problem into a multi-class problem with one multi-class classifier is trained on all unique label combinations found in the training data.
\nfrom skmultilearn.problem_transform import LabelPowerset\n# initialize label powerset multi-label classifier\nclassifier = LabelPowerset(LogisticRegression())\n\n# train\nclassifier.fit(x_train, y_train)\n\n# predict\npredictions = classifier.predict(x_test)Please refer my notebook multilabelnlpclassifier.ipynb from my repo for more details.
\nThe two types of email you can send and receive are plain text emails (any email that contains just plain old text with no formatting) and HTML emails, these are formatted and styled using HTML and inline CSS.\nHTML email is the use of HTML to provide formatting and semantic markup capabilities in an email that are not available in plain text.
\nAn HTML email is designed just like a website with the help of graphics, table columns, colors and links. A non-programmer can also create it since email marketing services provide flexible campaign builders. Email client vendors have not been as developing as web browser vendors in adopting new standards.
\nDefinition
\nEmails which are formatted using Hypertext Markup Language(HTML), as opposed to plain text email.
\nHow to Create an HTML Email
\nMany tools that create and send an email will offer pre-formatted, already built HTML templates that allow you to design emails without knowing or accessing any code of back-end.
\nThe best way to understand any process is to do it yourself, from level zero. We make any changes in the email editor; those changes will be automatically coded into the final result. This email building tool is the best option if you don't have an email designer, but you still want to send any professional marketing emails.
\nIf you want more control over the code of your emails and you are comfortable with HTML(that is just basic and easy), most email tools will allow you to import HTML files directly for using it as custom email templates. They have a wide variety of free HTML email templates available on the internet, and if you are familiar with HTML, it is a straightforward process to use that template in the email building tool of your own choice.
\nTo create an HTML email from scratch, you will need to have advanced knowledge of HTML. Because creating an HTML email from scratch can be quite tricky, we recommend you to work with a developer for this process, or you may go with a template for an easy process.
\nIf you choose to code your HTML email by hand, these are the necessary steps you need to use while creating HTML email:
\nHow to send HTML emails through Outlook?
\nYou can check HTML Email Template here:
\n\n","frontmatter":{"date":"October 09, 2020","updated_date":null,"description":"If you choose to code your HTML email by hand, there are many different things you need to use while creating HTML email.","title":"HTML Email Concept","tags":["Html","Email"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.7699115044247788,"src":"/static/68cacf6be60ee3e735f59ee70958946b/ee604/email_picture.png","srcSet":"/static/68cacf6be60ee3e735f59ee70958946b/69585/email_picture.png 200w,\n/static/68cacf6be60ee3e735f59ee70958946b/497c6/email_picture.png 400w,\n/static/68cacf6be60ee3e735f59ee70958946b/ee604/email_picture.png 800w,\n/static/68cacf6be60ee3e735f59ee70958946b/05d05/email_picture.png 1080w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Nivedita Singh","github":"Nivedita967","avatar":null}}}},{"node":{"excerpt":"These days we have all come across one of the coolest buzzwords in the IT industry: \"The Blockchain\". It might seem to be a new magic word…","fields":{"slug":"/engineering/blockchain-the-new-technology-of-security-trust/"},"html":"These days we have all come across one of the coolest buzzwords in the IT industry: \"The Blockchain\". It might seem to be a new magic word in the market that companies spell interest in their businesses. However, the complexity of it is incredibly far-reaching. Blockchain integrates the openness and flexibility of the internet with the security of cryptography to come out with a safer, faster way of verification of information and most importantly establishes trust in this open world.
\nBlockchain was first developed by an anonymous programmer or group of programmers known by a name 'Santoshi Nakamoto'.It was an underlying technology for the Bitcoin, which is used for peer-to-peer transactions. Blockchain at its heart is a list of transactions like a distributed ledger open to all in the network. It stores the data in such a way that it seems virtually impossible to add, update or remove any information stored without the notice of other users in a peer-to-peer network.
\nBlockchain mainly performs two tasks: collect and order data in blocks- similar to the traditional computer database and then chain them securely using cryptography.
\nLet us take a closer look at each block in this enormous chain –
\nThe copy of the complete blockchain is with all the participants so that they can detect tampering if the hash matches across the chain then everyone knows that it is trustworthy.\nBlockchain is an emerging technology but has been evolving ever since its innovation. This technology has the unlimited potential to bring about an upheaval in the way everyone- organisations, governments, individuals work together. It promises a simple, secure, paperless path to establish trust for virtual transactions of money, products and other confidential information worldwide.
\nBlockchain is one of the technologies that has gained popularity from its very birth, and now it is being used in many fields.
\nIn financial markets, trade is very dynamic where there is the exchange of money, assets involving multiple banks; this may lead to unexpected errors. To reduce this bottleneck blockchain came up with the idea of smart contracts which is a small computer program that describes the transactions step by step combining multiple blockchains, multiple assets and executes the transactions securely.
\nBlockchain can keep track of many commercial transactions and efficiently hold sensitive information. A digital id via blockchain secures the data stored and can be used worldwide in your fingertips.
\nBlockchain can be very handy in monitoring the supply chain in food and manufacturing industries by removing paper-based trails and also removing intermediaries between producers to customers. Not just the above applications it is used in many more places and has changed, is going to change the world around us.
\nThough it is a new technology, it has an enormous ability to transform everything existing now. As a coin has its two faces, blockchain technology also has some glitches as it can destroy the middlemen in many of the industries.\nBlockchain has already spread its root firmly in soils of the new world, and the swarm of transformation has already begun. It is the responsibility of all the young generation to make complete usage of this technology as it matures and make it to become a huge money plant.
\n","frontmatter":{"date":"October 08, 2020","updated_date":null,"description":"Learn about Blockchain technology and how it works.","title":"Blockchain: The new technology of trust","tags":["Blockchain","Cyber Security"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.6666666666666667,"src":"/static/39f6130fcc45e64f3049c5db8f8a19be/14b42/CoverPage.jpg","srcSet":"/static/39f6130fcc45e64f3049c5db8f8a19be/f836f/CoverPage.jpg 200w,\n/static/39f6130fcc45e64f3049c5db8f8a19be/2244e/CoverPage.jpg 400w,\n/static/39f6130fcc45e64f3049c5db8f8a19be/14b42/CoverPage.jpg 800w,\n/static/39f6130fcc45e64f3049c5db8f8a19be/a6352/CoverPage.jpg 960w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Shraddha V Prasad","github":"shraddhavp","avatar":null}}}}]},"markdownRemark":{"excerpt":"Google has prepared a roadmap to restrict third-party cookies in Chrome. Since 04 January 2024, Chrome has rolled out third-party cookie…","fields":{"slug":"/engineering/identity-impact-of-google-chrome-thirdparty-cookie-restrictions/"},"html":"Google has prepared a roadmap to restrict third-party cookies in Chrome. Since 04 January 2024, Chrome has rolled out third-party cookie restrictions for 1% of stable clients and 20% of Canary, Dev, and Beta clients.
\nWhat does it mean for user authentication?
\nOn one hand, Google believes third-party cookies are widely used for cross-site tracking, greatly affecting user privacy. Hence, Google wants to phase out (or restrict) supporting third-party cookies in Chrome by early Q2 2025 (subject to regulatory processes).
\nOn the other hand, Google introduced Privacy Sandbox to support the use cases (other than cross-site tracking and advertising) previously implemented using third-party cookies.
\nIn this article, we’ll discuss:
\nThird-party cookie restrictions affect user authentication in three ways, as follows.
\nIf your website or app uses an external Identity Provider (IdP) — like LoginRadius, the IdP sets a third-party cookie when the user authenticates on your app.
\nIf you have multiple apps across domains within your organization and authentication is handled using an IdP (internal or external) with web SSO, you already use third-party cookies to facilitate seamless access for each user using a single set of credentials.
\nIf you have implemented web SSO with one primary domain and multiple sub-domains of the primary domain, third-party cookie restrictions may not apply. For now, Google doesn’t consider the cookies set by sub-domains as third-party cookies, although this stance may change in the future.
\nFor example, you have apps at example.com, travel.example.com, stay.example.com, and web SSO is handled by auth.example.com. In this case, third-party cookie restrictions don’t apply.
Federated SSO is similar to, albeit different from, web SSO. It can handle multiple IdPs and applications—aka., Service Providers (SPs)—spanning multiple organizations. It can also implement authentication scenarios that are usually implemented through web SSO.
\nUsually, authentication is handled on a separate pop-up or page when the user wants to authenticate rather than on the application or website a user visits.
\nFor example, you already use federated SSO if you facilitate authentication for a set of apps through multiple social identity providers as well as traditional usernames and passwords.
\n\n\nNote: It is also possible to store tokens locally, not within cookies. In this case, third-party cookie restrictions won’t affect token-based authentication. However, the restrictions still affect authentication where tokens are stored within third-party cookies (a common and secure method).
\n
Google has been developing alternative features and capabilities for Chrome to replace third-party cookies as part of its Privacy Sandbox for Web initiative.
\nSpecific to authentication, Google recommends the following:
\nCHIPS are a restricted way of setting third-party cookies on a top-level site without making them accessible on other top-level sites. Thus, they limit cross-site tracking and enable specific cross-site functionalities, such as maps, chat, and payment embeds.
\nFor example, a user visits a.com with a map embed from map-example.com, which can set a partitioned cookie that is only accessible on a.com.
If the user visits b.com with a map embed from map-example.com, it cannot access the partitioned cookie set on a.com. It has to create a separate partitioned cookie specific to b.com, thus blocking cross-site tracking yet allowing limited cross-site functionality.
You should specifically opt for partitioned cookies (CHIPS), which are set with partitioned and secure cookie attributes.
\nIf you’re using an external identity provider for your application, CHIPS is a good option to supplant third-party cookie restrictions.
\nHowever, CHIPS may not be ideal if you have a web SSO or federated SSO implementation. It creates separate partitioned cookies for each application with a separate domain, which can increase complexity and create compatibility issues.
\nWith Storage Access API, you can access the local storage in a third-party context through iframes, similar to when users visit it as a top-level site in a first-party context. That is, it gives access to unpartitioned cookies and storage.
\nStorage Access API requires explicit user approval to grant access, similar to locations, camera, and microphone permissions. If the user denies access, unpartitioned cookies and storage won’t be accessible in a third-party context.
\nIt is most suitable when loading cross-site resources and interactions, such as:
\nVerifying user sessions when allowing interactions on an embedded social post or providing personalization for an embedded video.\nEmbedded documents requiring user verification status to be accessible.
\nAs it requires explicit user approval, it is advisable to use Storage Access API when you can’t implement an identity use case with the other options.
\nWith Related Website Sets, you can declare a primary website and associatedSites for limited purposes to grant third-party cookie access and local storage for a limited number of sites.
Chrome automatically recognizes related website sets declared, accepted, and maintained in this open-source GitHub repository: Related Website Sets
\nIt provides access through Storage Access API directly without prompting for user approval, but only after the user interacts with the relevant iframe.
\nIt is important to declare a limited number of domains in related website sets that are meaningful and used for specific purposes. Google may block or suspend any exploitative use of this feature.
\nThe top-level site can also request approval for specific cross-site resources and scripts to Storage Access API using resuestStorageAccessFor() API.
If you’re using an external identity provider for your web application, you can declare the domain of the identity provider in the related set to ensure limited third-party cookies and storage access to the identity provider, thus ensuring seamless user authentication.
\nRelated Website Sets can also work to supplement third-party cookie restrictions in web SSO and federated SSO if the number of web applications (or domains) is limited.
\nFedCM API enables federated SSO without third-party cookies.
\nWith FedCM API, a user follows these steps for authentication:
\nYou can access a user demo of FedCM here: FedCM.
\nFor more information about implementing federated SSO with FedCM API, go through the FedCM developer guide.
\nFirstly, we’re committed to solving our customers' user identity pain points — and preparing for the third-party cookies phase-out is no different.
\nWe’ll implement the most relevant and widely useful solutions to facilitate a smooth transition for our customers.
\nPlease subscribe to our blog for more information. We’ll update you on how we help with the third-party cookie phase-out.
\nThe proposed changes to phase out third-party cookies and suggested alternatives are evolving as Google has been actively collaborating and discussing changes with the border community.
\nMoreover, browsers like Firefox, Safari, and Edge may approach restricting third-party cookies differently than Google does.
\nFrom LoginRadius, we’ll keep you updated on what we’re doing as a leading Customer Identity and Access Management (CIAM) vendor to prepare for the third-party cookie phase-out.
\nTop-level site: It is the primary site a user has visited.
\nFirst-party cookie: A cookie set by the top-level site.
\nThird-party cookie: A cookie set by a domain other than the top-level site. For example, let’s assume that a user has visited a.com, which might use an embed from loginradius.com to facilitate authentication. If loginradius.com sets a cookie when the user visits a.com, it is called a third-party cookie as the user hasn’t directly visited loginradius.com.