In the whole software development process, QA testing has a unique space. QA is responsible for ensuring that developed software is bug-free and works concerning technical and business requirements.
\nQA engineers must have a good understanding of each project and what it means to accomplish. They must deliver quality software to the clients.
\nGiven the responsibility of a QA's job, it is natural to face many challenges in their day-to-day tasks. This article will help to understand the most common of those challenges faced by any QA.
\nUsually, QA teams face unstable environment setup issues that we need to prepare for most of what we have. Sometimes the server gets stuck due to overload and requires a restart many times during testing etc.
\nEscalate these issues to the seniors and make sure you get the environment ready for the testing.
\nNow and then, we realize that a tool is not the right choice for the project. We do not have any other option but to keep using it because the clients/organization already have licenses and would not go for a new one until the current license expires.
\nIt is not fun, but you learn alternate options. Or at least, one can conclude with regards to if the possibilities work.
\nThe biggest challenge QA is to receive requests for last-minute testing. The primary reasons for such demands are that the development process takes more time than expected and the time for testing is underestimated. Generally, testing and debugging take 50% of the development time. When QA has a short time frame for verification, they should check software against the main business specifications. Software testing should begin at least three days of release.
\nIn the case of QA, it's faster to create a document from scratch than to use the one created by others. Using test cases created by others increases the time of verification and puts limits as far as discovering bugs.
\nEverybody thinks of the successful releases of new features or products, but the reality could be different. From our testing experience, the software doesn't typically release from the first time in most cases. The best time for releasing the software is the start of the week. Thus it gives development and QA teams the rest of the week to manage with whatever comes up.
\nWhile trying to make an accurate estimate, some software estimations could be entirely unpredictable and go wrong. As developers, QA also doesn't have 100% security from unexpected issues.
\nDevelopers and QA engineers should work closely. Testing should be done once part of the development process is done, and after that, bug fixing activity should start. QA submit a test report and only after that debugging begins.
\nIt is slightly common to change project requirements mid-sprint in agile development projects. While this can be frustrating for the team and due to that testers can be affected. They may need to re-try the whole extent of testing since even the littlest changes to a codebase should be gone through various tests to guarantee its steadiness and similarity with existing code.
\nNaturally, last-minute changes in the requirement can be difficult for testers to deal with, especially if there are tight deadlines to deliver results.
\nProfessional differences between development and testing teams are still common. Developers think that testing is a final process of the software development life cycle, and testers do not require anything apart from a list of user journeys and technical requirements.
\nHowever, testers may have difficulty identifying flaws in the code if they are not acquainted with the development process. If they do not understand how the software works, they will have trouble creating test cases to find all possible bugs.
\nResolving the challenges mentioned above will not only make the lives of QA testers much easier. Still, it will also streamline the software development process to make it more useful and time-efficient. By making it easy for QAs to do their job well, organizations can ensure that their products meet all business requirements and function in the best possible way.
\n","frontmatter":{"date":"March 05, 2021","updated_date":null,"title":"Top 9 Challenges Faced by Every QA","tags":["QA","Processes","Testing"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.9047619047619047,"src":"/static/db8a52fc54f8db8b8c3c7c58719160a3/ee604/testing-challenges.png","srcSet":"/static/db8a52fc54f8db8b8c3c7c58719160a3/69585/testing-challenges.png 200w,\n/static/db8a52fc54f8db8b8c3c7c58719160a3/497c6/testing-challenges.png 400w,\n/static/db8a52fc54f8db8b8c3c7c58719160a3/ee604/testing-challenges.png 800w,\n/static/db8a52fc54f8db8b8c3c7c58719160a3/f3583/testing-challenges.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Neha Vyas","github":"nehavyasqa","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/qa-teams-deliver-quality-software/"},"html":"Being a Quality Analyst, it's our responsibility to run test cycles on the system and push releases every two weeks to ensure the system is updated with the latest fixes and everything is bug free on the live platform. Usually, every release includes amendments as well as fixes done. If we talk about the resources, the team typically has 8-10 resources, including developers and the QA team.
\nTherefore, we should always be proactive about every release and avoid any stress during the release. Everything has to be managed properly when it comes to execution from the development end or testing from the QA end. For that purpose, we have to follow proper strategies to ensure that everything is getting tested and the release we approve met the expected quality standards.
\nWe have penetrated limits in two ways. In this day and age, we are a client confronting unit, as we get with our clients straightforwardly about their issues which they experience over the software or what are the highlights which they need to join on the stage. Also, we are actively participating in the design discussions or any inputs required from our end, which we receive from the client's end, to make sure we are going in the right direction as per the client's vision. Our testing experience always helps detect any flaw or defects before doing any coding work, which reduces additional efforts and minimizes the work. This way, we can ensure everything is done based on the client's given timeline and a quality outcome.
\nAs it's impractical to test everything each time at whatever point you push any delivery. For that reason, you need to focus on the piece of code that has been chipped away. This way, you can test the primary work component without postponing things and afterward run a total test cycle to ensure no other part has been clashed by the changes. For this reason, there is a new delivery check with the group of what piece of code has been refreshed and what are the results, and generally center around that part to run tests. This way, you can focus on the work and endeavors which should be put.
\nWe center around those pieces of the code and utilize existing computerization tests to deal with different parts. If you realize something worked in the final delivery and you're not contacting it in this delivery, at that point, you don't have to invest an excessive amount of time in testing. So base your delivery standards on the new code that is being added.
\nQA has to make sure that whatever work needs to be done is appropriately prioritized. For example, if particular area of a platform is not affecting things much and can be taken care at later stage then it should be on the lowest priority; If less than one percent of our users are on a particular browser, issues specific to that browser stand out enough to be noticed. And the area which is affecting the whole system should be done on the very first priority. There should be priorities defined like – Blocker, Emergency, Highest, High, Normal, and Low! So always work as per the critical attention on a specific feature and consider the target user demands. What is their feedback or issues to which they want to have access? If something moved beyond clients and we find bugs, we need to fix them in the following release.
\nYou will often hear from the development team that, \"…. It works fine with me; there must be some network issue or system issue on your end\". We have to make sure not to have such conflicts in between because, in the end, our goal has to be – Delivering bug free solution!
\nFor that purpose, Our QA and development teams run their tests in the same environment. As our builds move through the development pipeline, we should test the code under the production environment to build our staging environment to simulate our customer's production environments.
\nAlways plan to arrange a dedicated performance team who can run tests as soon as a product is steady and tell the team about new versions and requirements to evaluate the performance risks.
\nTo update performance teams with all the latest information and provide them with an environment close to production as you can. To make sure that there shouldn't be any gaps between and can appropriately test the product.
\nQA team runs regression tests in the last phase of the product cycle, and it is that process that confirms the product delivery. If any of the regression tests fail, QA informs the managers/stakeholders so that appropriate actions can be taken for further release as it's never acceptable to deliver the product if any of the tests fail, which might end up in more significant problems for the team as well as the client. So the QA team always makes sure to run regression tests to verify everything properly.\nWe additionally automate our regression cycle, so it just requires a couple of days to run.
\nAs we maintain customer data in our database collections, we need to guarantee that it should remain attuned with any new versions that have been released or about to release. Whenever we release a new version, we run several updates to check that no data was harmed, and in case we find any data-corrupting bugs, those become our highest priority. We also perform manual backward compatibility testing while taking steps towards finding an automated and more efficient approach. Nonetheless, it would be best to do some manual testing, as this is one of the last stages before production.
\nWe perform post-release sanity tests on our production account to authorize that everything works as projected, including all third-party systems. We initially perform tests utilizing our existing production account but then create a new account to check that the process will continue to work correctly as new customers register.
\nThe best testing practices should inculcate all other processes in general and risk management processes in particular in them. The focus should be to improve the software's overall quality while aiming to reduce the cost with continuous monitoring during and after the software release. While doing quality assurance testing, the tester should comply with all the fundamental principles and industry practices and look at the product from the end user's perspective.
\n","frontmatter":{"date":"March 01, 2021","updated_date":null,"title":"QA Testing Process: How to Deliver Quality Software","tags":["QA","Processes","Testing"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/5fe40fa61d401b955af6527da8353680/2a4de/software-testing.png","srcSet":"/static/5fe40fa61d401b955af6527da8353680/69585/software-testing.png 200w,\n/static/5fe40fa61d401b955af6527da8353680/497c6/software-testing.png 400w,\n/static/5fe40fa61d401b955af6527da8353680/2a4de/software-testing.png 600w","sizes":"(max-width: 600px) 100vw, 600px"}}},"author":{"id":"Neha Vyas","github":"nehavyasqa","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/best-hacking-tools/"},"html":"Ever since you enter the sphere of ethical Hacking, you usually need to get your hand on most of the Hacking Tools. The more tools you recognize, the more your hacking career is getting ready to fly off. Therefore let's discuss some of the necessary hacking tools.\n
\n
It is a free and open-source tool that's used for network discovery and security auditing.\nNmap is a powerful tool because it is often used to scan vast networks having thousands of machines. It's a command-line tool. Nmap suite additionally includes a complicated GUI that's referred to as \"ZenMap\".\nIt supports a large variety of OS that is:
\nIt uses raw IP packets to determine
\nIt is essentially a Security Assessment and Penetration Testing tool. Metasploit is often used to launch an attack on alternative systems with it.\nIt uses a vulnerable system on that security testing may be conducted to use the failings within the system.\n
\nMetasploit may enforce as follows:
\n
It is an open-source tool that's used to capture traffic on the network. It's essentially a network protocol analyzer tool.\n
\nWireshark helps in:
\nIt additionally captures HTTP packet transmission over the network. Click on \"Follow protocol connection\" within the HTTP packet. Currently, you'll be able to see the username and passwords that are captures over the network.\n
\n\n
JTR is free and open-source software that's wide employed by hackers for password cracking. It uses the varied cryptanalytics attacks like \"Dictionary Attack\" and \"Brute-Force Attack\".\nIt additionally comes with the business version moreover, i.e., \"John the Ripper Professional.\" It's a lot of accessible versions providing a lot of practicality in password cracking at the enterprise level.\n
\nJohn the Ripper working:
\nThe rate at which the password is going to be cracked depends utterly on the password's strength and offered wordlist. It keeps attempting to break the password continuously till the termination command isn't given.\n
\nJohn the ripper download link\n
\n
It is an integrated platform that's used for activity a check on net application security.\nIt provides a large variety of tools that are used from initial mapping to exploiting the applications' vulnerabilities. Once the issues are detected, hackers will use it to break into the security of the system.\n
\nBurp Suite comes in 3 editions:
\nBurp Suite features:
\n
It is one of the quickest IP addresses and port scanner. By exploitation, this hacker will gather data concerning open ports within the target system.\nIt pings every IP address within the target system to see whether it's active or not. Further, it resolves the hostnames and determines the MAC address.\n
\nFeatures:
\nAngry IP Scanner download link\n
\nIt is a web-server assessment tool. It is an open-source platform that performs tests against web servers to seek multiple vulnerable files, misconfigurations, out-of-date servers, and programs on its web server.\nIt depends on HTTP response to seeing whether or not a page or script exists on the target.\n
\nFeatures:
\nWriting a foolproof code is hard, especially with a team collaborating on a single project. Thus it becomes increasingly important to ensure that no new code breaks existing functionality. And for this purpose, automated tests are used.
\nIn this blog, we will learn how to integrate Jest and Puppeteer in our project for End-to-End Testing. Jest is a javascript testing framework maintained by Facebook. Puppeteer is a Node library created by Google, which provides a high-level API to control headless Chrome. Before we delve deeper into Jest and Puppeteer, let's familiarise ourselves with a few things.
\nEnd-to-End(E2E) Testing: E2E testing refers to the testing of a complete functionality of some application. It checks that a functionality acts as intended.
\nHeadless Testing: Headless testing is a way or running browser UI tests without any browser UI. It is a preferred method mainly due to its performance. It is fast, light-weight, and less resource-intensive.
\nTo install Jest and Puppeteer, open the command line in your project directory and run:
\nFor npm users:
\nnpm install --save-dev jest puppeteer jest-puppeteer
For yarn users:
\nyarn add --dev jest puppeteer jest-puppeteer.
Now that we've installed Jest and Puppeteer, it is time to set up the testing environment. Create a file jest.config.js in the project root directory and write the following code in it:
module.exports = {\n preset: "jest-puppeteer",\n globals: {\n URL: "http://localhost:8080"\n },\n testMatch: [\n "<path-to-the-tests-folder>/**.test.js"\n ],\n verbose: true\n};Defining the preset sets up a puppeteer environment for testing. Test match directs it to the folder where tests are defined, for a piece of more detailed knowledge about running Jest with Puppeteer check this out.
\nNow create a file jest-puppeteer.config.js in the project root directory. Here we define the testing environment further. To begin with you can add the following in this file:
module.exports = {\n launch: {\n headless: process.env.HEADLESS !== 'false',\n slowMo: process.env.SLOWMO ? process.env.SLOWMO : 0,\n devtools: true\n }\n}Now we can begin writing our tests. In this example, we'll start writing a basic test to check the title of the page opened. Now head over to the tests folder defined and create a file title.test.js. As you might have noticed, .test.js would be the file extension of the jest tests defined.
This is what a typical title.test.js would look like:
// Defining the timeout for the test\nconst timeout = process.env.SLOWMO ? 6000 : 4000;\nconst fs = require('fs');\n\n// Go to the specified path and wait for the domcontent to load before running the tests\nbeforeAll(async () => {\n path = fs.realpathSync('file://<file-path>');\n await page.goto('file://' + path, {waitUntil: 'domcontentloaded'});\n});\n\ndescribe('Title of the page', () => {\n test('Title of the page', async () => {\n // Gets page title\n const title = await page.title();\n // Compares it with the intended behaviour\n expect(title).toBe('<title-of-the-page>');\n\n }, timeout);\n});Hurray! You've added Headless UI tests to your project. Now you can similarly define tests for different functionalities using the docs here.
\nIt could give a sandbox-ing error if you run it in a container without a defined user. When running headless Chrome in a container without a defined user, the chromeOptions environment property needs a --no-sandbox args (in addition to the other headless args), or Chrome won't be able to startup. The possible solutions to this are:
\nargs: ['--no-sandbox'] in jest-puppeteer.config.js (not recommended).sudo apt-get install gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget Happy coding! =)
\n","frontmatter":{"date":"October 15, 2020","updated_date":null,"title":"End-to-End Testing with Jest and Puppeteer","tags":["Testing","Jest","Puppeteer"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.6129032258064515,"src":"/static/bc9e7eb8f88f913bb3f3ba67edf01319/01a28/jest-puppeteer.png","srcSet":"/static/bc9e7eb8f88f913bb3f3ba67edf01319/69585/jest-puppeteer.png 200w,\n/static/bc9e7eb8f88f913bb3f3ba67edf01319/497c6/jest-puppeteer.png 400w,\n/static/bc9e7eb8f88f913bb3f3ba67edf01319/01a28/jest-puppeteer.png 799w","sizes":"(max-width: 799px) 100vw, 799px"}}},"author":{"id":"Shreya Sharma","github":"cypherean","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/unit-testing/"},"html":"A lot has been written and talked about Unit Testing in the IT industry for the last couple of years. I have heard and discussed the same over the previous 2.5 decades of my IT career.
\nEven though a lot is talked about and is done in this field but still today, every company is struggling with this. But before we proceed with the above situation, let us first understand its importance.
\nA strong foundation is the basis of a strong building in the same way a proper unit testing is the foundation of high-quality software.
\nUnit Testing, if done correctly, ensures that each of the building blocks of the software product is robust and will make high-quality software when put together.
\nUnit testing is an essential tool in any serious software developer's toolbox. However, writing a good unit test for a specific piece of code can often be very hard.
\nDevelopers frequently believe that their struggles are triggered by a lack of essential testing expertise or hidden unit testing techniques after experiencing trouble testing their own or someone else's code.
\nTesting of the unit requires testing individual components of the programme or application software. The primary objective behind this is to verify that all the individual components function as expected.
\nAs the smallest possible part of the software that can be evaluated, a unit is known. It usually has a couple of inputs and a single output.
\nBoth testers and developers can isolate each module, detect and repair device defects at a very early stage of the life cycle of software development with this testing method (SDLC).
\nBy considering stubs, mock artifacts, drivers, and unit testing frameworks, this methodology effectively helps validate the consistency of a section of code.
\nBecause it is practiced at the initial testing level, this testing methodology ensures that the vulnerabilities are detected and corrected at the early stage of SDLC, even before they become costly for businesses to repair when they are later identified.
\nThe developers and testers can help save time with a suitable unit testing practice as bugs can be found early in the process as it is the initial testing step. Skipping or restricting the unit testing practice will adversely increase the defects, and repairing them later becomes difficult.
\nTherefore before preparing for the integration testing, it is essential to practice unit testing at the initial stage of the software testing process.
\nWell, it is not that a developer loves to find bugs in his/her coded code. Still, when it comes to unit testing, they at times(most of the time) see it has an additional overhead on them, especially when one has to follow and stick to a lot of protocols, frameworks, and documentation, etc.
\nAlso, it is commonly misunderstood that finding issues is the QC team's problem. We have to code. In my view, it is incorrect thinking as Quality Software is standard or combines commitment from a company and it's the team.
\nHere are a few benefits of unit testing:
\nThe most significant advantage of unit testing is this. You would need to make improvements to the old design and code when you add more functionality to any programme, which can be costly and risky. If you use the technique of unit testing, this will save a lot of time and can make the whole process much quicker and simpler.
\nUnit testing increases code consistency dramatically. It allows designers to recognize the smallest faults present in the units before they go for integration testing.
\nAt a very early point, unit testing helps detect all sorts of problems with the programme. Until going any further, software developers should then focus on those problems first.
\nThis is the primary benefit of this because no other program component is compromised as the issues are fixed at an early stage. This results in improved performance, decreased downtime, and lower costs that would otherwise occur due to the entire design process's stalling.
\nWhen you test each part of the program individually, refactoring the code or upgrading the device library becomes much simpler. If there are any issues, they are found early on, and it thus becomes much easier to make improvements to the system.
\nBefore it goes on to the next level, the accuracy of each unit is checked. This implies that until it's incorporated with other companies, the device is proven to be in good working order.
\nUnit verification considers the whole system's documentation. By reading each module's documentation, developers who want to learn about the features of a specific programme or framework will quickly learn about the device.
\nIt makes it possible for them to have a detailed understanding of the system and what each part does.
\nTo a great degree, unit testing will simplify the debugging process. If a particular test fails, it is essential to debug only the most recent improvements made to the code.
\nIn the early stages, any problems or glitches in the system are detected by unit testing, and the cost of bug fixes is significantly reduced because of this. If these vulnerabilities are found later, so repairing them would be even more costly.
\nIt is important for the unit testers to get a step-by-step instructional document in order to practice unit testing with the manual testing process.
\nHowever, automated unit testing is often chosen by most companies, taking into account the efforts necessary for manual testing.
\nIn the end, remember that I am the first and the most basic foundation in a big building of Quality and have to always keep the same in mind whenever I code since no one is interested in buying a well-packaged product if its ingredients are not of high quality.
\nBy doing the above I am not only contributing to high-quality software but I can devote later high bug-fixing time to things like improving the performance/security of my code and avoiding going back to the drawing board for issues related to performance or security etc.
\nAlways remember in the software development unit testing serves as \"A stitch in time saves nine\".
\n","frontmatter":{"date":"September 29, 2020","updated_date":null,"title":"Unit Testing: What is it and why do you need it?","tags":["Unit Testing","QA","Testing"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/35dc4816d23601e5b75a44da15e3d18c/14b42/unit-testing.jpg","srcSet":"/static/35dc4816d23601e5b75a44da15e3d18c/f836f/unit-testing.jpg 200w,\n/static/35dc4816d23601e5b75a44da15e3d18c/2244e/unit-testing.jpg 400w,\n/static/35dc4816d23601e5b75a44da15e3d18c/14b42/unit-testing.jpg 800w,\n/static/35dc4816d23601e5b75a44da15e3d18c/47498/unit-testing.jpg 1200w,\n/static/35dc4816d23601e5b75a44da15e3d18c/0e329/unit-testing.jpg 1600w,\n/static/35dc4816d23601e5b75a44da15e3d18c/947df/unit-testing.jpg 5472w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Sudhanshu Pandey","github":null,"avatar":null}}}},{"node":{"fields":{"slug":"/engineering/big-data-testing-strategy/"},"html":"Earlier, we were only dealing with well-structured data, hosted in large data warehouses, and investing a cost in maintaining those data warehouses and hiring expert professional to maintain and secure information hosted in that data warehouse. Data was structured and can be queried anything as per the needs. But now, this exponential growth of data generates a new vision for data science along with some major challenges.
\nBig Data is something which is growing exponentially with time, and carry raw but very valuable information inside that can change the future of any enterprise. It is a collection, which represents a large dataset, may be collected from multiple sources, or stored in an organization.\nLet‟s understand a real-time example, Big companies like Ikea and Amazon are leveraging the benefit of big data by collecting data from customer‟s buying patterns at their stores, their internal stock information, and their inventory demand-supply relations and analyze all, in seconds even in real-time to add value to its customer experience.
\nSo, extracting information from a large dataset somewhat calls a concept of Data mining which is an analytic process originally designed to explore large datasets. The ultimate goal of data mining is to search for consistency in a pattern or systematic relationship between variables, which helps in predicting the next pattern or behavior.
\nNow, if we take concepts of data mining forward along with large data set, to some extent it becomes a blocker for our existing approach, because big data may contain structured or unstructured data even it may contain data in multiple formats also.
\nTesting is an art of achieving quality in your software product, in terms of perfection of functionality, performance, user experience, or usability. But for big data testing, you need to keep your focus more on the functional and performance aspects of an application. Performance is the key parameter in any big data application which is meant to process terabytes of data. Successful processing of terabytes of data using a commodity cluster with a number of other supportive components needs to be verified. Processing should be faster and accurate which demands a high level of testing.
\nProcessing may be of three types:–
\nAnd based on which, we need to integrate different components along with NoSQL data store as per the needs.
\nData plays a vital role in the testing of big data applications. Application is meant to process data and provide an expected output based on implemented logic. The logic needs to be verified before moving to production, as the implementation of logic is completely based on business requirements and data.
\n1. Test Data Quality
\nGood quality test data is as important as the test environment. In the big data world, data can have any format or size, it may be in the form of a document, XML, JSON, PDF, etc. at the same time data size may go up to terabytes of petabytes. Hence, test data should also have multiple formats and size should be large enough to ensure the handling of large data processing. In big data testing, it needs data with logical values as per the application requirement and format which is supported by the application.
\nAlong with it, data quality is another aspect of big data testing. Ensuring the quality of data before processing through application ensures the accuracy of the final output. Data quality testing itself is a huge domain and covers a lot of best practices which include – data completeness, conformity, accuracy, validity, duplication, and consistency, etc. It should be included in the big data testing and this ensures the level of accuracy application is supposed to provide.
\n2. Test Data Generation
\nThe generation of test data is again a challenging job, there are multiple parameters, which have to be taken care of while generating test data. It needs a tool, which can help to generate data and should have functions or logic can also be applied over it. Tools like Talend (an open studio) is the best candidate to fulfill the requirements of data generation.
\n3. Data Storage
\nAfter the generation of test data along with quality, it needs to host on a file system. For testing big data applications, data should be stored in the system similar to the production environment. As we are working in big data space, there should have a different number of nodes, and data must be in a distributed environment.
\nIn Big data testing, the test environment should be efficient enough to process a large amount of data as done in the case of a production environment. Real-time production environment clusters generally have 30-40 nodes of cluster and data is distributed on the cluster nodes. There must have some minimum configuration for each node used in the cluster. A cluster may have two modes, in-premise or cloud. For testing in big data, it needs the same kind of environment with some minimum configuration of node.
\nScalability is also desired to be there in the test environment of big data testing, it helps to study the performance of application with the increase in the number of resources. That data can be used to define SLA (service level agreement) for that particular application.
\nBig Data Testing can be categorized into three stages:
\nStep 1: Data Staging Validation
\nThe first stage of big data testing, also known as a Pre-Hadoop stage, is comprised of process validation.
\nStep 2: “Map Reduce” Validation
\nValidation of “Map Reduce” is the second stage. Business logic validation on every node is performed by the tester. Post that authentication is done by running them against multiple nodes, to make sure that the:
\nStep 3: Output Validation Phase
\nThe output validation process is the final or third stage involved in big data testing. The output data files are created and they are ready to be moved to an EDW (Enterprise Data Warehouse) or any other such system as per requirements. The third stage consisted of:
\nBig data applications are meant to process a large amount of data, and it is expected that it should take minimum time to process maximum data. Along with it, application jobs should consume a considerable amount of memory and CPU. In big data testing, performance parameter plays an important role and helps to define SLA's. It covers the performance of the base machine and cluster. Also, for example, In the case of Hadoop, map-reduce jobs should be written with proper coding guidelines, to perform better in the production environment. Profiling can also be done on map-reduce jobs before integration, to ensure their optimized execution.
\nIn big data testing, certain challenges are involved which needs to be addressed by the big data testing approach.
\n1. Test Data
\nExponential growth had been observed in the growth of data in the last few years. A huge amount of data are being generated daily and stored in large data centers or data marts. So, there is a demand for efficient storage and a way to process it in an optimized way. If we consider the telecom industry, it generates a large number of call logs daily and they need to be processed for better customer experience and compete in the market. The same goes with the test data, test data should be similar to production data and should contain all the logically acceptable fields in it.
\nThis becomes a challenge for testing big data application, generating test data similar to production data is a real challenge. Test data should also be large enough to verify proper working big data application.
\n2. Environment
\nThe processing of data highly depends on the environment and its performance. An optimized environment setup gives high performance and fast data processing results. Distributed computing is used for the processing of big data which has data hosted in a distributed environment. The testing environment should have multiple numbers of nodes and data should be distributed over the nodes. At the same time, it also needs to monitor those nodes, to ensure the highest performance with minimum CPU and memory utilization. Nodes should be monitored and there should have a graphical presentation of node performance. So, the test environment has two aspects – distributed nodes and their monitoring, which should be covered in the testing approach.
\n3. Performance
\nPerformance is the key requirement of any big data application, and of course because of which enterprises are moving towards NoSQL technologies, technologies that can handle their big data and process in the minimum time frame. A large dataset should be processed in a minimum considerable time frame. In big data testing, performance testing is a challenge, it requires monitoring of cluster nodes during execution and also time is taken for every iteration of execution.
\nBig Data is the trend that is revolutionizing society and its organizations due to the capabilities it provides to take advantage of a wide variety of data, in large volumes and with speed. Keeping the challenges in mind we have defined the approach of testing big data applications. This approach of big data testing will make it easy for a test engineer to verify and certify the business requirement implementations and for stack holders, it saves a huge amount of cost, which has to be invested to get the expected business returns.
\n","frontmatter":{"date":"September 02, 2020","updated_date":null,"title":"Big Data - Testing Strategy","tags":["Testing","Big Data","Big Data Testing"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.7699115044247788,"src":"/static/370acda0a8a364ef7098ce2df0537d12/ee604/big-data.png","srcSet":"/static/370acda0a8a364ef7098ce2df0537d12/69585/big-data.png 200w,\n/static/370acda0a8a364ef7098ce2df0537d12/497c6/big-data.png 400w,\n/static/370acda0a8a364ef7098ce2df0537d12/ee604/big-data.png 800w,\n/static/370acda0a8a364ef7098ce2df0537d12/f3583/big-data.png 1200w,\n/static/370acda0a8a364ef7098ce2df0537d12/5707d/big-data.png 1600w,\n/static/370acda0a8a364ef7098ce2df0537d12/eeb1b/big-data.png 1920w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Sudhey Sharma","github":"sudheysharma","avatar":null}}}}]}},"pageContext":{"tag":"Testing"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}