Cloud Cost Optimization is the method of reducing the total cloud spending by finding mismanaged capital and scale-up of Right Sizing computing services. CCO only charge for the services you use, the cloud gives companies infinite scalability and lower IT costs.
\nOn-Premise Infrastructure was only option around 10 years ago but managing On-premise Infrastructure include the extra cost with other challenges, Challenges are Network, Hardware, Cooling, Power, Space also needed technical staff for taking care all infrastructure
\nThe Colocation is when a business places its own server in a third-party data center and uses its infrastructure services. Colocation takes responsibility for Network, Hardware, Cooling, Space also business not need to worry about technical man force for managing servers
\nStill business have more challenges OS software, Application security, Data storage
\nThe Cloud service provider is providing solutions for all the challenges. The cloud service provider takes total responsibility for hardware, network, space, power, maintaining, and security also they are managing all technical workforce for the whole infrastructure
\n Cloud infrastructure is increasing day by day. Choosing the right cloud provider is the most important decision for long term success. The big three cloud providers are Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). These three cloud providers are providing mostly services so they might be confused about choosing the right cloud provider. We can easily evaluate the right cloud provider from the following steps All Big Cloud Providers are providing discounted Instances or spot instances. Spot instances are unused instances. Cloud Providers offered up to 90% discount on these instances compared to on-demand or reserved instances. The majority of organizations have some workloads that are not critical, We can reduce the cost for not critical workload by using spot instances. AWS, Azure, and Google (GCP) all provide the option to use Spot Instances. Identify the right size for the Instance, not an easy task. We need to configure multiple matrices in our cloud service provider. Key metrics to look for are CPU and memory usage. Identify instances with a maximum CPU usage and memory usage of the month. Generally, people make mistakes during checking metrics. Most of the people prefer averages in monitoring but averages mislead the measurement. Let understand this by example You are running an online technology tutorial site. Suppose 1million active users on your website in normal days. Now you are hosting weekly technology webinars in your platform. During webinars time your platform traffic should increase. At that time CPU usages and memory usages should be high compared to the rest of the time. Now suppose you have checked the 24 hours average CPU usages and memory usages. It was around 40% and you have decided this is underutilized and scale down the lower size instance. It can impact you 5% - 10% traffic Let’s look at a real-world example. This chart shows the overall CPU usages First Image showing Average CPU utilization. You can see Maximum CPU was 18.6 % Now, let’s check the CPU 99th percentile:\n As expected, the 99th percentile is higher than the average. 99th percentile is around 93% The conclusion is We need to always check the 99th percentile. The average can mislead and it can impact your users. We can create different types of resources and applications utilization monitoring matrix and based on the data we can configure multiple alarms. You can configure notification or alarms on email, SMS from the Cloud provider dashboard. you can also configure alarms that automatically stop or terminate EC2 instances or VM when instance unused or underutilized according to the configured threshold. During the development or some POC as a developer or devops person we have to create some instances or resources but sometimes we forgot to terminate the instances or resources. You can minimize the this extra cost by creating a group of alarms that sends an email notification to developers whose instances have been underutilized or ideal for some hours, then terminates an instance. It will save the overall infra cost. different cloud providers provide different ways for creating these type alarms You can save around 75% cost for your Non - Production Development, Staging, and QA environment. Non - Production environment generally needed during working days. You can turn off these servers during off-hours. Cost-saving depends on the infrastructure size, it can be hundred, thousands of dollars You can create automation scripts for infrastructure deployment. Schedule the script in your cloud provider Application in-memory cache reduces the cost of transferring data in the network and overall application performance because reduce the traffic between the database servers or any other external application reduce the network level cost in the cloud also caching improves the efficiency and accessibility of data that used repeatedly or frequently accessed. Suppose your application is fetching user configuration or settings in every request from the database server. You can keep this type of configuration, which is not changing frequently in the in-memory cache. it will save a lot network-level cost Data Transfer cost mostly hidden or Some time we don't take care of it. Generally, data transfer is free in the same region between different services Storage, Compute service, etc.\nIf you do a lot of cross-region transfer, it will increase your network data transfer cost also if you will deploy multiple services in the same region it will improve application performance This includes knowing what you spend in detail, how specific services are billed, and the ability to display how (or why) you spent a specific amount Here, keep in mind key capabilities such as the ability to create shared accountability, hold frequent cost reviews, analyze trends, and visualize the impact of your actions on a near-real-time basis. You can also use cost controls like budget alerts and quotas to keep your costs in check over time. Enterprises or Mid Level companies are adopting multi-cloud infrastructure. Consider this recent prediction from IDC: “By 2020, over 90% of enterprises will use multiple cloud services and platforms.” Or this one from 451 Research: “The future of IT is multi-cloud and hybrid with 69% of respondents planning to have some type of multi-cloud environment by 2019.” Organizations need to develop a cost optimization culture and awareness. Cost optimization is an ongoing activity in the organization. Need to decide someone responsible for the cost optimization it can be an Engineering team or DevOps team. Most cloud providers provide billing alarm’s they can alert you in case of cost increment also we can configure budget in the Cloud provider dashboard. This post will cover a demo working setup of a service mesh architecture using Envoy using a demo application. In this service mesh architecture, we will be using Envoy proxy for both control and data plane. The setup is deployed in a Kubernetes cluster using Amazon EKS. We will be deploying an echo-grpc test application provided by Google in their article related to gRPC load balancing and was used as a reference to test the service mesh setup with Envoy. The article covers setting up Envoy as an edge proxy only.\nThis is a simple gRPC application that exposes a unary method that takes a string in the content request field and responds with the content unaltered.\nRepo: grpc-gke-nlb-tutorial Configuration of envoy for the sidecar deployment: envoy-echo.yaml: A couple things to note here. Run:\n Deployment of echo-grpc application with 3 replicas. The config contains two containers, one for application and another being the Envoy image. Here, echo-grpc is test application and envoy is being deployed in the same pod. Config volumes are mounted so that the envoy can read the configmaps. Run:\n We are using headless service for echo-grpc. Using service as headless will expose the Pods IP to the DNS server of kubernetes which will be used by Envoy to do service discovery for the pods. echo-service.yaml In the above config file, we are exposing two ports, one for envoy sidecar (this is the same port we mentioned in the config map of sidecar envoy) and one for the service itself. Run:\n Creating a service of type LoadBalancer so that client can access the backend service. envoy-service.yaml: Run:\n Since we are deploying front envoy LoadBalancer on port 443, we have to create a self-signed certificate to make it terminate SSL/TLS connection. Configuration for the edge Envoy: envoy-configmap.yaml Since we will be offloading HTTPS, we are using port_value of 443. Most of the configurations are same as of sidecar envoy except for three things: Run:\n envoy-deployment.yaml Run:\n Proto file for the echo-grpc service: ccho.proto: Run the following command to call the server:\n The output will be similar to something like this: Run the above command multiple times and check the value of the hostname field every time which will contain the pod name of one of the 3 pods deployed. Kafka Streams is a Java library developed to help applications that do stream processing built on Kafka. To learn about Kafka Streams, you need to have a basic idea about Kafka to understand better. If you’ve worked with Kafka before, Kafka Streams is going to be easy to understand. Kafka Streams is a streaming application building library, specifically applications that turn Kafka input topics into Kafka output topics. Kafka Streams enables you to do this in a way that is distributed and fault-tolerant, with succinct code. Stream processing is the ongoing, concurrent, and record-by-record real-time processing of data. Let us get started with some highlights of Kafka Streams: There are two individual processors in the topology: There is actually a close relationship between streams and tables, the so-called stream-table duality Let's Start with the Setup using Scala instead of Java. The Kafka Streams DSL for Scala library is a wrapper over the existing Java APIs for Kafka Streams DSL. To Setup things, we need to create a You can create a topic using the below commands (need to have Kafka pre installed) StreamsBuilder provide the high-level Kafka Streams DSL to specify a Kafka Streams topology. Creates a KStream from the specified topics. Store the input stream to the output topic. Starts the Streams Application You can send data to the input topic using And can fetch the data from the output topic using You can add the necessary dependencies in your build file for sbt or pom file for maven. Below is an example for build.sbt. Let us modify the code a little bit to try out a WordCount example:\nThe code splits the sentences into words and groups by word as a key and the number of occurences or count as value and is being sent to the output topic by converting the KTable to KStream. With the above process, we can now implement a simple streaming application or a word count application using Kafka Streams in Scala. If you want to set up kafka on Windows, here is a quick guide to implement apache kafka on windows OS. You’re probably hosting your MongoDB on a reliable cloud service provider say Atlas for instance because you really want to focus on your idea and delegate all the subtle key management areas such as networking, storage, access, etc. It all looks good initially until your small idea starts turning into a business and the cost starts skyrocketing. Even if that is not the case, this post will still give you a general overview of the technical complexities involved (and bucks saved!) if you were to migrate to a self-hosted solution. BTW, how much savings are we talking about? Let’s do a quick comparison between an Atlas instance and a self-hosted MongoDB on AWS. Atlas (~$166/month) $0.0416/hour for the instance and additional pricing based on the EBS type and storage (~ calculator.aws) It is almost 4.5x savings just in terms of the infrastructure! Now that you know the major why(s) and are still reading this post, without further ado, let’s dive into the tech. Since the entire content can be a bit exhausting in one place, I’m going to divide this into 2 related posts. I’m going to mention below the guide for setting up an instance running RedHat Enterprise Linux 8 on AWS. This is because MongoDB generally performs better with the xfs file-system. I’ve used a It is recommended that your DB should have access to at least 1GB of RAM and 2 real cores as that directly affects the performance during caching & concurrency mechanisms as handled by the default engine WiredTiger. You can read more about the production notes related to the RAM and CPU requirements here. Configuration Overview: I’m assuming that you’re familiar with creating a VM on AWS. Now, once the instance is in running state, assign an Elastic IP to it and then simply do a remote login into the machine. We'll need the Elastic IP to setup public hostname for the instance We have added 2 additional EBS volumes other than the Root FS for Data and Logs which are yet to be mounted (Remember the 30Gb and 3Gb?). You can list the volume blocks using, The additional volumes will be listed right after the root block (refer to the arrows) Copy the UUIDs listed for /dev/xvdb and /dev/xvdc. Refer to the “LABEL” for block identification Now open the The hostname will be used to identify your database server on the network. You can either use the above assigned Elastic IP or Domain name (if available). Open the This is optionally required in order to control the maximum number of acceptable connections while keeping the system stable.\nOpen the Now that all of the infra related prerequisites are sorted, reboot the instance, and let’s proceed to MongoDB installation. Create a file Now let’s install MongoDB. MongoDB by default uses the following paths to store the data and the internal logs: /var/lib/mongo → Data Create the directories Change user & group permissions The mongod daemon/service must be first running before we proceed to create a user.\nLet’s use the default config(stored in The above command will start the mongod daemon in fork mode (default). Now, let’s login to the server and create our first admin user. Open a mongo shell Use the \"admin\" database to create the root-admin Create the admin user Create a regular user Shutdown the server for now. We'll restart again with the modified config Here, we’ll enable the database authentication and modify the bind-address for our server to be accessible in the public domain.\nOpen Save the config and restart the server. You can verify if the credentials work using, That is it about the initial setup! Please stay tuned for my next related post on the detailed migration process and tips on keeping the DB production-ready. P.S. Thanks to Piyush Kumar for helping curate this post! We're delightfully Announcing the upcoming releases of LoginRadius Identity-Experience-Framework Open-Source Automation These full-version releases will include more test coverage and major changes with several improvements & code optimizations, the details have been given below. CI/CD pipelines for automation is to make everyday development tasks simple that can save time and reduce human intervention to check them every single code commit. We are not just delivering automation we deliver values to our customers, some useful tips \"to working with selenium and nightwatch\" can be expected with this release Although current configuration does not support different browsers, still you can run on any browser by making few changes in nightwatch.json detailed guide is here It is agreed that we are committed to delivering you a seamless experience in automation, but still, if you face any issue or need any enhancements, please report us here Today we will learn about the progressive web app, a very interesting and trending topic in current times, We will start with the basic steps first on how Progressive Web App works with vanilla js. Also, we’ll cover the concept of PWAS without using any of the frameworks, it's just a plain A Progressive Web app is an application which is built on top of HTML, JS and CSS with some extra features like Various big companies are moving towards the progressive web app instead of native apps, as it gives some much flexibility to end-users. In some websites you can find the + icon at the right side of the address bar, those are progressive web apps. We will be using Once you fill up those details your final output of the After that, we need to install the So finally our application is ready to serve in localhost. Now let's add some Create a directory for the app and add js, css, and images subdirectories. It should look like this when you’re finished: You will understand the use of When writing the code for Progressive Web Application we need to keep in mind two requirements For this app, to handle the first requirement we will add the static text in the Html page and for the second requirement, we will use the viewport. So let's add this code inside the Next, we need to create a file Once you add the code save all the files and run the script As we can see our application is running in the browser, let test it using the LightHouse Extension in Google Chrome browser. For the best result, I recommend testing in an incognito tab of google chrome. As you can see we have green signals for both of our above requirement: But still, there is lots of work to do, so let's start with the first feature of PWA's. Let's create a new file called Now update the We should also declare the theme colour which should match with your manifest.json file as Once you run the lighthouse test again you can see we got green signals in some points like So let's add the And the index.html will head can will look like You can also verify the After running the test again, we can see all the The next requirement of our app will be the service worker. The service worker is a script that runs in the background without any user interaction. The task of service worker will be to download and cache our content and then serve it when the user is offline. To understand more on service workers you can check this doc from mozilla. To integrate service workers we need to create a file called Note: Service worker will only have the access to the files of the current and sub-directories hence we need to place it in the root of our project The first 2 variables Note: Here I have used the Finally, we will load our service worker, for that create And add this script in the So the final version of our Now our application is made as fully PWA supported, you can verify it by running the lighthouse test. We are done with the code, now the last thing remaining in to host it in any of the You can get the full source code of this example in this Github repo I hope you enjoyed the progressive web apps using vanilla js tutorial and found it useful. Please let me know what you think in the comments below. Google has prepared a roadmap to restrict third-party cookies in Chrome. Since 04 January 2024, Chrome has rolled out third-party cookie restrictions for 1% of stable clients and 20% of Canary, Dev, and Beta clients. What does it mean for user authentication? On one hand, Google believes third-party cookies are widely used for cross-site tracking, greatly affecting user privacy. Hence, Google wants to phase out (or restrict) supporting third-party cookies in Chrome by early Q2 2025 (subject to regulatory processes). On the other hand, Google introduced Privacy Sandbox to support the use cases (other than cross-site tracking and advertising) previously implemented using third-party cookies. In this article, we’ll discuss: Third-party cookie restrictions affect user authentication in three ways, as follows. If your website or app uses an external Identity Provider (IdP) — like LoginRadius, the IdP sets a third-party cookie when the user authenticates on your app. If you have multiple apps across domains within your organization and authentication is handled using an IdP (internal or external) with web SSO, you already use third-party cookies to facilitate seamless access for each user using a single set of credentials. If you have implemented web SSO with one primary domain and multiple sub-domains of the primary domain, third-party cookie restrictions may not apply. For now, Google doesn’t consider the cookies set by sub-domains as third-party cookies, although this stance may change in the future. For example, you have apps at Federated SSO is similar to, albeit different from, web SSO. It can handle multiple IdPs and applications—aka., Service Providers (SPs)—spanning multiple organizations. It can also implement authentication scenarios that are usually implemented through web SSO. Usually, authentication is handled on a separate pop-up or page when the user wants to authenticate rather than on the application or website a user visits. For example, you already use federated SSO if you facilitate authentication for a set of apps through multiple social identity providers as well as traditional usernames and passwords. Note: It is also possible to store tokens locally, not within cookies. In this case, third-party cookie restrictions won’t affect token-based authentication. However, the restrictions still affect authentication where tokens are stored within third-party cookies (a common and secure method). Google has been developing alternative features and capabilities for Chrome to replace third-party cookies as part of its Privacy Sandbox for Web initiative. Specific to authentication, Google recommends the following: CHIPS are a restricted way of setting third-party cookies on a top-level site without making them accessible on other top-level sites. Thus, they limit cross-site tracking and enable specific cross-site functionalities, such as maps, chat, and payment embeds. For example, a user visits If the user visits You should specifically opt for partitioned cookies (CHIPS), which are set with partitioned and secure cookie attributes. If you’re using an external identity provider for your application, CHIPS is a good option to supplant third-party cookie restrictions. However, CHIPS may not be ideal if you have a web SSO or federated SSO implementation. It creates separate partitioned cookies for each application with a separate domain, which can increase complexity and create compatibility issues. With Storage Access API, you can access the local storage in a third-party context through iframes, similar to when users visit it as a top-level site in a first-party context. That is, it gives access to unpartitioned cookies and storage. Storage Access API requires explicit user approval to grant access, similar to locations, camera, and microphone permissions. If the user denies access, unpartitioned cookies and storage won’t be accessible in a third-party context. It is most suitable when loading cross-site resources and interactions, such as: Verifying user sessions when allowing interactions on an embedded social post or providing personalization for an embedded video.\nEmbedded documents requiring user verification status to be accessible. As it requires explicit user approval, it is advisable to use Storage Access API when you can’t implement an identity use case with the other options. With Related Website Sets, you can declare a Chrome automatically recognizes related website sets declared, accepted, and maintained in this open-source GitHub repository: Related Website Sets It provides access through Storage Access API directly without prompting for user approval, but only after the user interacts with the relevant iframe. It is important to declare a limited number of domains in related website sets that are meaningful and used for specific purposes. Google may block or suspend any exploitative use of this feature. The top-level site can also request approval for specific cross-site resources and scripts to Storage Access API using If you’re using an external identity provider for your web application, you can declare the domain of the identity provider in the related set to ensure limited third-party cookies and storage access to the identity provider, thus ensuring seamless user authentication. Related Website Sets can also work to supplement third-party cookie restrictions in web SSO and federated SSO if the number of web applications (or domains) is limited. FedCM API enables federated SSO without third-party cookies. With FedCM API, a user follows these steps for authentication: You can access a user demo of FedCM here: FedCM. For more information about implementing federated SSO with FedCM API, go through the FedCM developer guide. Firstly, we’re committed to solving our customers' user identity pain points — and preparing for the third-party cookies phase-out is no different. We’ll implement the most relevant and widely useful solutions to facilitate a smooth transition for our customers. Please subscribe to our blog for more information. We’ll update you on how we help with the third-party cookie phase-out. The proposed changes to phase out third-party cookies and suggested alternatives are evolving as Google has been actively collaborating and discussing changes with the border community. Moreover, browsers like Firefox, Safari, and Edge may approach restricting third-party cookies differently than Google does. From LoginRadius, we’ll keep you updated on what we’re doing as a leading Customer Identity and Access Management (CIAM) vendor to prepare for the third-party cookie phase-out. Top-level site: It is the primary site a user has visited. First-party cookie: A cookie set by the top-level site. Third-party cookie: A cookie set by a domain other than the top-level site. For example, let’s assume that a user has visited Choose the Right Cloud Provider
\n\n
\nDiscounted Instances
\n\n
\nIdentify Instance Right Size
\nHow to Check Metrics
\nCreating Underutilization Alarm
\n\n
\nCreating time based actions
\nIn-memory cache Storage
\nData Transfer Cost Optimization
\nCost visibility
\nConsider a Multi-Cloud Architecture
\nBenefits of multi-cloud Architecture
\n\n
\nConclusion
\nPre-requisites
\n\n
\nkubectl create namespace envoygo get github.com/fullstorydev/grpcurlSidecar Deployment
\n
\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: envoy-echo\ndata:\n envoy.yaml: |\n static_resources:\n listeners:\n - address:\n socket_address:\n address: 0.0.0.0\n port_value: 8786\n filter_chains:\n - filters:\n - name: envoy.http_connection_manager\n config:\n access_log:\n - name: envoy.file_access_log\n config:\n path: "/dev/stdout"\n codec_type: AUTO\n stat_prefix: ingress_https\n route_config:\n name: local_route\n virtual_hosts:\n - name: https\n domains:\n - "*"\n routes:\n - match:\n prefix: "/api.Echo/"\n route:\n cluster: echo-grpc\n http_filters:\n - name: envoy.health_check\n config:\n pass_through_mode: false\n headers:\n - name: ":path"\n exact_match: "/healthz"\n - name: "x-envoy-livenessprobe"\n exact_match: "healthz"\n - name: envoy.router\n config: {}\n clusters:\n - name: echo-grpc\n connect_timeout: 0.5s\n type: STATIC\n lb_policy: ROUND_ROBIN\n http2_protocol_options: {}\n load_assignment:\n cluster_name: echo-grpc\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: "127.0.0.1"\n port_value: 8081\n health_checks:\n timeout: 1s\n interval: 10s\n unhealthy_threshold: 2\n healthy_threshold: 2\n grpc_health_check: {}\n admin:\n access_log_path: "/dev/stdout"\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 8090\n
\n*, allowing all domains to pass-through.kubectl apply -f envoy-echo.yaml -n envoy
\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: echo-grpc\nspec:\n replicas: 3\n selector:\n matchLabels:\n app: echo-grpc\n template:\n metadata:\n labels:\n app: echo-grpc\n spec:\n containers:\n - name: echo-grpc\n image: <hub-username>/echo-grpc\n imagePullPolicy: Always\n resources: {}\n env:\n - name: "PORT"\n value: "8081"\n ports:\n - containerPort: 8081\n readinessProbe:\n exec:\n command: ["/bin/grpc_health_probe", "-addr=:8081"]\n initialDelaySeconds: 1\n livenessProbe:\n exec:\n command: ["/bin/grpc_health_probe", "-addr=:8081"]\n initialDelaySeconds: 1\n - name: envoy\n image: envoyproxy/envoy:v1.9.1\n resources: {}\n ports:\n - name: https\n containerPort: 443\n volumeMounts:\n - name: config\n mountPath: /etc/envoy\n volumes:\n - name: config\n configMap:\n name: envoy-echokubectl apply -f echo-deployment.yaml -n envoyHeadless Service Configuration
\n
\napiVersion: v1\nkind: Service\nmetadata:\n name: echo-grpc\nspec:\n type: ClusterIP\n clusterIP: None\n selector:\n app: echo-grpc\n ports:\n - name: http2-echo\n protocol: TCP\n port: 8786\n - name: http2-service\n protocol: TCP\n port: 8081kubectl apply -f echo-service.yaml -n envoyFront Envoy Configuration
\n
\napiVersion: v1\nkind: Service\nmetadata:\n name: envoy\nspec:\n type: LoadBalancer\n selector:\n app: envoy\n ports:\n - name: https\n protocol: TCP\n port: 443\n targetPort: 443Creating self-signed certificates
\nkubectl apply -f envoy-service.yaml -n envoy\n
\nkubectl describe svc/envoy -n envoynslookup <your load balancer aadess>openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout privkey.pem -out cert.pem -subj \"/CN=<ip-address>\"kubectl create secret tls envoy-certs --key privkey.pem --cert cert.pem --dry-run -o yamlEdge Envoy configuration
\n
\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: envoy-conf\ndata:\n envoy.yaml: |\n static_resources:\n listeners:\n - address:\n socket_address:\n address: 0.0.0.0\n port_value: 443\n filter_chains:\n - filters:\n - name: envoy.http_connection_manager\n config:\n access_log:\n - name: envoy.file_access_log\n config:\n path: "/dev/stdout"\n codec_type: AUTO\n stat_prefix: ingress_https\n route_config:\n name: local_route\n virtual_hosts:\n - name: https\n domains:\n - "*"\n routes:\n - match:\n prefix: "/api.Echo/"\n route:\n cluster: echo-grpc\n http_filters:\n - name: envoy.health_check\n config:\n pass_through_mode: false\n headers:\n - name: ":path"\n exact_match: "/healthz"\n - name: "x-envoy-livenessprobe"\n exact_match: "healthz"\n - name: envoy.router\n config: {}\n tls_context:\n common_tls_context:\n tls_certificates:\n - certificate_chain:\n filename: "/etc/ssl/envoy/tls.crt"\n private_key:\n filename: "/etc/ssl/envoy/tls.key"\n clusters:\n - name: echo-grpc\n connect_timeout: 0.5s\n type: STRICT_DNS\n lb_policy: ROUND_ROBIN\n http2_protocol_options: {}\n load_assignment:\n cluster_name: echo-grpc\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: echo-grpc.envoy.svc.cluster.local\n port_value: 8786\n health_checks:\n timeout: 1s\n interval: 10s\n unhealthy_threshold: 2\n healthy_threshold: 2\n grpc_health_check: {}\n admin:\n access_log_path: "/dev/stdout"\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 8090\n
\nkubectl apply -f envoy-configmap.yaml -n envoyDeployment Configuration
\n
\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: envoy\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: envoy\n template:\n metadata:\n labels:\n app: envoy\n spec:\n containers:\n - name: envoy\n image: envoyproxy/envoy:v1.9.1\n resources: {}\n ports:\n - name: https\n containerPort: 443\n volumeMounts:\n - name: config\n mountPath: /etc/envoy\n - name: certs\n mountPath: /etc/ssl/envoy\n readinessProbe:\n httpGet:\n scheme: HTTPS\n path: /healthz\n httpHeaders:\n - name: x-envoy-livenessprobe\n value: healthz\n port: 443\n initialDelaySeconds: 3\n livenessProbe:\n httpGet:\n scheme: HTTPS\n path: /healthz\n httpHeaders:\n - name: x-envoy-livenessprobe\n value: healthz\n port: 443\n initialDelaySeconds: 10\n volumes:\n - name: config\n configMap:\n name: envoy-conf\n - name: certs\n secret:\n secretName: envoy-certskubectl apply -f envoy-deployment.yaml -n envoyTesting
\n
\nsyntax = "proto3";\n\npackage api;\n\nservice Echo {\n rpc Echo (EchoRequest) returns (EchoResponse) {}\n}\n\nmessage EchoRequest {\n string content = 1;\n}\n\nmessage EchoResponse {\n string content = 1;\n}grpcurl -d '{\"content\": \"echo\"}' -proto echo.proto -insecure -v <load_balancer_or_external_ip>:443 api.Echo/Echo
\nResolved method descriptor:\nrpc Echo ( .api.EchoRequest ) returns ( .api.EchoResponse );\n\nRequest metadata to send:\n(empty)\n\nResponse headers received:\ncontent-type: application/grpc\ndate: Wed, 27 Feb 2019 04:40:19 GMT\nhostname: echo-grpc-5c4f59c578-wcsvr\nserver: envoy\nx-envoy-upstream-service-time: 0\n\nResponse contents:\n{\n "content": "echo"\n}\n\nResponse trailers received:\n(empty)\nSent 1 request and received 1 responseReferences
\n\n","frontmatter":{"date":"July 06, 2020","updated_date":null,"description":null,"title":"Service Mesh with Envoy","tags":["Service Mesh","Envoy","Microservices"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.834862385321101,"src":"/static/f5fbebaefd091eb3f33c225fa71b0814/14b42/front-image.jpg","srcSet":"/static/f5fbebaefd091eb3f33c225fa71b0814/f836f/front-image.jpg 200w,\n/static/f5fbebaefd091eb3f33c225fa71b0814/2244e/front-image.jpg 400w,\n/static/f5fbebaefd091eb3f33c225fa71b0814/14b42/front-image.jpg 800w,\n/static/f5fbebaefd091eb3f33c225fa71b0814/47498/front-image.jpg 1200w,\n/static/f5fbebaefd091eb3f33c225fa71b0814/0e329/front-image.jpg 1600w,\n/static/f5fbebaefd091eb3f33c225fa71b0814/6ed45/front-image.jpg 2100w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Piyush Kumar","github":"kpiyush17","avatar":null}}}},{"node":{"excerpt":"Kafka Streams is a Java library developed to help applications that do stream processing built on Kafka. To learn about Kafka Streams, you…","fields":{"slug":"/engineering/stream-processing-using-kafka/"},"html":"What are Kafka Streams?
\nWhat is Stream processing?
\n\n
\nKafka Streams Concepts:
\n\n
\n\n
\n\n
\nImplementing Kafka Streams
\nKafkaStreams Instance. It needs a topology and configuration (java.util.Properties). We also need a input topic and output topic. Let's look through a simple example of sending data from an input topic to an output topic using the Streams API
\nkafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic inputTopic\nkafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic outputTopic
\nval config: Properties = {\n val properties = new Properties()\n properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "your-application")\n properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")\n properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")\n properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE)\n properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String())\n properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String())\n properties\n }
\nval builder: StreamsBuilder = new StreamsBuilder
\nval inputStream: KStream[String,String] = builder.stream(inputTopic, Consumed.`with`(Serdes.String(), Serdes.String()))
\ninputStream.to(outputTopic)(producedFromSerde(Serdes.String(),Serdes.String())
\nval kEventStream = new KafkaStreams(builder.build(), config)\nkEventStream.start()\nsys.ShutdownHookThread {\n kEventStream.close(10, TimeUnit.SECONDS)\n }
\nkafka-console-producer --broker-list localhost:9092 --topic inputTopic
\nkafka-console-consumer --bootstrap-server localhost:9092 --topic outputTopic --from-beginning
\n// Kafka\nlibraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"\nlibraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1" artifacts( Artifact("javax.ws.rs-api", "jar", "jar")) // this is a workaround. There is an upstream dependency that causes trouble in SBT builds.
\nval textLines: KStream[String, String] = builder.stream[String, String](inputTopic)\nval wordCounts: KTable[String, Long] = textLines\n\t\t.flatMapValues(textLine => textLine.toLowerCase.split("\\\\W+"))\n\t\t.groupBy((_, word) => word)\n\t\t.count()(materializedFromSerde(Serdes.String(),Serdes.Long()))\n\twordCounts.toStream.to(outputTopic)(producedFromSerde(Serdes.String(),Serdes.Long())\n
\n
\nOutline
\n\n
\n1. Setting up the Infrastructure
\nSpin up an EC2 Instance
\nt3.small instance that comes with 2 vCPUs and 2Gb of RAM albeit you can select any instance of your choice.\n
\n\n
\n\n
\n
\n$ ssh -i <PEM_FILE> ec2-user@<ELASTIC_IP>Mount Additional Volumes
\n
\n$ sudo lsblk
\n$ sudo blkid/etc/fstab file and paste the configuration in the following format.
\nUUID=<COPIED_UUID_FOR_DATA> /var/lib/mongo xfs defaults,nofail 0 0\nUUID=<COPIED_UUID_FOR_LOGS> /var/log/mongodb xfs defaults,nofail 0 0Update Hostname
\n/etc/hostname file and append the entry. For e.g.
\nip-xx.us-east-2.compute.internal **<ELASTIC_IP> <DOMAIN_1> <DOMAIN_2>** ...Update the Process Limits (Optional)
\n/etc/security/limits.conf file and add the following entries.
\n* soft nofile 64000\n* hard nofile 64000\n* soft nproc 32000\n* hard nproc 32000
\n1. Setting up MongoDB
\nAdd the Repo Source
\n/etc/yum.repos.d/mongodb-org.4.2.repo and add the following package repository details.
\n[mongodb-org-4.2]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
\n$ sudo yum -y install mongodb-orgCreate directories and setup permissions
\n\n
\n
/var/log/mongodb → Logs
\n$ sudo mkdir /var/lib/mongo\n$ sudo mkdir /var/log/mongodb
\n$ sudo chown mongod:mongod /var/lib/mongo\n$ sudo chown mongod:mongod /var/log/mongodCreate an Admin User
\n/etc/mongod.conf) for now and start the daemon process.
\n$ sudo -u mongod mongod -f /etc/mongod.conf
\n$ mongo
\n> use admin
\n> db.createUser({user: "admin", pwd: "password", roles: [{role: "root", db: "admin"}]})
\n> db.createUser({user: "normal_user", pwd: "password", roles: [{role: "readWriteAnyDatabase", db: "admin"}]})
\n> db.shutDownServer()Setting up Authentication
\n/etc/mongod.conf and make the below changes.
\n# network interfaces\nnet:\n port: 27017\n bindIp: 0.0.0.0 # accessible on the network address\nsecurity:\n authorization: enabled # creds will be required for making db operations
\n$ sudo -u mongod mongod -f /etc/mongod.confTest Login
\n
\n$ mongo -u admin -p passwordRoadmap of IDX-auto-tester
\n
\nThe below changes are planned to go live in multiple scheduled major releases starting with mid of June 2020. Release Roadmap:
\nJuly 2020:
\nTest Cases:
\n\n
\n\n
\nAugust 2020:
\nUsability Improvements:
\n\n
\nDocumentations:
\n\n
\nheadless mode\n
\nSeptember 2020:
\nCode Level Improvements and Optimizations:
\n\n
\nBetter Reporting:
\n\n
\n\n
\nOctober 2020:
\nOvercoming the Limitations:
\n\n
\noptional and disabled email verification flows.multifactor-authenticationvanillaJS.What are Progressive Web Apps?
\nservice worker and manifest.json file. These extra features will give the ability to run the web application offline.Setup your base app for PWA
\nnpm to start our project, so firstly create a root folder using this command mkdir first-pwa-app or another name of your choice then go to that directory by using this command cd first-pwa-app. Run npm init
\nnpm init\n\npackage name: (first-pwa-app)\nversion: (1.0.0)\ndescription: This is the first PWA example\nentry point: (index.js): index.html\ntest command:\ngit repository:\nkeywords:\nauthor: Mohammed Modi\nlicense: (ISC)package.json file will be below
\n{\n "name": "first-pwa-app",\n "version": "1.0.0",\n "description": "This is the first PWA example",\n "main": "index.html",\n "scripts": {\n "test": "echo \\"Error: no test specified\\" && exit 1"\n },\n "author": "Mohammed Modi",\n "license": "ISC"\n}serve dependency to serve the application in localhost so install it using npm npm i serve --save. And then add a script in the package.json file.
\n{\n ...\n "scripts": {\n "test": "echo \\"Error: no test specified\\" && exit 1",\n "serve": "serve ."\n },\n ...\n}html and cssSetting up the Javascript
\n
\n/first-pwa-app # Project Folder\n /css # Stylesheets\n /js # Javascript\n /images # Image files.\n package.json\n package-lock.json\n index.html\n manifest.json\n sw.jsmanifest.json and sw.js files once we move ahead in this tutorial.Add Some code for app interface
\n\n
\nindex.html file.
\n<!DOCTYPE html>\n<html lang="en">\n <head>\n <meta charset="utf-8" />\n <title>My First PWA</title>\n <link rel="stylesheet" href="css/style.css" />\n <meta name="viewport" content="width=device-width, initial-scale=1.0" />\n </head>\n <body class="fullscreen">\n <div class="container">\n <h1 class="title">My First PWA!</h1>\n </div>\n </body>\n</html>css/style.css and paste this code inside this
\nbody {\n font-family: sans-serif;\n}\n\n/* Make content area fill the entire browser window */\n/* Make content area fill the entire browser window */\nhtml,\n.fullscreen {\n display: flex;\n height: 100%;\n margin: 0;\n padding: 0;\n width: 100%;\n}\n\n/* Center the content in the browser window */\n.container {\n margin: auto;\n}\n\n.title {\n font-size: 3rem;\n}npm run serve which will start your application in http://localhost:5000/Testing the App
\n\n
\nAdding a Web App Manifest
\nmanifest.json in the root of our application. To understand more about the manifest file check out [this doc]
\n{\n "name": "My First Progressive Web app",\n "short_name": "First PWA",\n "lang": "en-US",\n "start_url": "/index.html",\n "display": "standalone",\n "background_color": "white",\n "theme_color": "white"\n}index.html head tag by adding the manifest.json file as a link
\n<head>\n ...\n <link rel="manifest" href="/manifest.json" />\n ...\n</head>meta tag.
\n<head>\n ...\n <meta name="theme-color" content="white" />\n ...\n</head>\n
\npwa-icon-256.png and pwa-icon-256.png for this demo you can find in this repo. You can use various sizes in it. And finally, add the favicon.png in the root of the project.icons attribute after the short-name
\n{\n "name": "My First Progressive Web app",\n "short_name": "First PWA",\n "icons": [\n {\n "src": ".images/pwa-icon-256.png",\n "sizes": "256x256",\n "type": "image/png"\n },\n {\n "src": ".images/pwa-icon-512.png",\n "sizes": "512x512",\n "type": "image/png"\n }\n ],\n "lang": "en-US",\n "start_url": "/index.html",\n "display": "standalone",\n "background_color": "white",\n "theme_color": "white"\n}
\n<head>\n ...\n <link rel="apple-touch-icon" href="images/pwa-icon-256.png" />\n ...\n</head>manifest.json file by opening Chrome DevTools, in the Application Tab, the first tab in the sidebar manifestmanifest.json related issues are solved, The remaining issues can be resolved by adding the service worker.Adding Service Worker
\nsw.js in the root of our project.\n
\n
\nlet cacheName = "my-first-pwa";\nlet filesToCache = ["/", "/index.html", "/css/style.css", "/js/main.js"];\n\n/* Start the service worker and cache all of the app's content */\nself.addEventListener("install", (e) => {\n e.waitUntil(\n caches.open(cacheName).then(function (cache) {\n return cache.addAll(filesToCache);\n })\n );\n});\n\n/* Serve cached content when offline */\nself.addEventListener("fetch", (e) => {\n e.respondWith(\n caches.match(e.request).then((response) => {\n return response || fetch(e.request);\n })\n );\n});cacheName and filesToCache are used to create an offline cache in the browser. cacheName is used to create a cache of specific name and filesToCache will be the list of files that we need to cache, / will store the root files that is index.html even if the user does not specify the file name in the URL and then all other files specified in the array.\n
\nfilesToCache array as a list of static files just to demonstrate the purpose of PWA, it will stop working in production if even one file fails to load.js/main.js file and copy the below code.
\nwindow.onload = () => {\n "use strict";\n\n if ("serviceWorker" in navigator) {\n navigator.serviceWorker.register("./sw.js");\n }\n};index.html file.
\n...\n<script src="js/main.js"></script>index.html file will look like:
\n<!DOCTYPE html>\n<html lang="en">\n <head>\n <meta charset="utf-8" />\n <title>Hello World</title>\n <link rel="apple-touch-icon" href="images/pwa-icon-256.png" />\n <link rel="stylesheet" href="css/style.css" />\n <link rel="manifest" href="/manifest.json" />\n <meta name="viewport" content="width=device-width, initial-scale=1.0" />\n <meta name="theme-color" content="white" />\n </head>\n <body class="fullscreen">\n <div class="container">\n <h1 class="title">Hello World!</h1>\n </div>\n <script src="js/main.js"></script>\n </body>\n</html>Finishing Up
\nhttps domain. As the PWA will be required to run in the secure domain.\n
\nHow is User Authentication Affected?
\nExternal Identity Providers
\nWeb SSO
\nexample.com, travel.example.com, stay.example.com, and web SSO is handled by auth.example.com. In this case, third-party cookie restrictions don’t apply.Federated SSO
\n\n
\nChrome’s Alternatives for Third-Party Cookies
\n\n
\nCookies Having Independent Partitioned State (CHIPS)
\na.com with a map embed from map-example.com, which can set a partitioned cookie that is only accessible on a.com. b.com with a map embed from map-example.com, it cannot access the partitioned cookie set on a.com. It has to create a separate partitioned cookie specific to b.com, thus blocking cross-site tracking yet allowing limited cross-site functionality.Storage Access API
\nRelated Website Sets
\nprimary website and associatedSites for limited purposes to grant third-party cookie access and local storage for a limited number of sites.resuestStorageAccessFor() API.Federated Credential Management (FedCM) API
\n\n
\nHow is LoginRadius Preparing for the Third-party Cookie Phase-out?
\nIn Conclusion
\nGlossary
\na.com, which might use an embed from loginradius.com to facilitate authentication. If loginradius.com sets a cookie when the user visits a.com, it is called a third-party cookie as the user hasn’t directly visited loginradius.com.References
\n\n","frontmatter":{"date":"July 08, 2024","updated_date":null,"description":"Google Chrome has planned to phase out third-party cookies, which will affect different website functionalities depending on third-party cookies. This blog focuses on how this phase-out affects identity and user authentication and discusses alternatives for overcoming challenges.","title":"How Chrome’s Third-Party Cookie Restrictions Affect User Authentication?","tags":["Identity","Cookies","Chrome"],"pinned":null,"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/eb7396060c0adc430dbed2d04b63d431/ee604/third-party-cookies-phaseout-chrome.png","srcSet":"/static/eb7396060c0adc430dbed2d04b63d431/69585/third-party-cookies-phaseout-chrome.png 200w,\n/static/eb7396060c0adc430dbed2d04b63d431/497c6/third-party-cookies-phaseout-chrome.png 400w,\n/static/eb7396060c0adc430dbed2d04b63d431/ee604/third-party-cookies-phaseout-chrome.png 800w,\n/static/eb7396060c0adc430dbed2d04b63d431/f3583/third-party-cookies-phaseout-chrome.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Raghunath Reddy","github":"raghunath-r-a","avatar":null}}}},"pageContext":{"limit":6,"skip":198,"currentPage":34,"type":"//engineering//","numPages":52,"pinned":"17fa0d7b-34c8-51c4-b047-df5e2bbaeedb"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}