Archive for: February, 2022

Community Innovation – Crowd Sourcing in Software Testing

Feb 17 2022 Published by dayat under Uncategorized

Many hands make software work

The stakes for Microsoft, which was outlining its Office 2010 product strategy, were extremely high. According to Microsoft’s earnings statements, Microsoft Office productivity suite generates more revenue than any other business division, says Gregg Keizer,who covers Microsoft and general technology news for Computerworld.

Months before Microsoft released Office 2010 productivity suite, 9 million people downloaded the beta version to test the software and to provide feedback. Through this program, Microsoft collected 2 million valuable comments and insights from those testers.

Denise Carlevato, a Microsoft usability engineer for 10 years, and her colleagues from Microsoft’s Virtual Research Lab observed how people used new features. Their objective was to make Microsoft Office fit the way millions of people used their product and to help them work better. It was a massive, controlled crowd sourcing project.

According to Carlevato, “That’s just what you have to do to cater to as broad an audience as possible”. This is not especially easy; to retain the loyalty of millions is a herculean task. “While the team was building it and giving insights and information, it was still all like a big jigsaw puzzle. Toward the end when this all came together I had the opportunity to go out and see how the web apps were actually going to live with real people.”
The scenario

Developing a new software product is always exciting, especially to watch ideas take form and truly become a reality. Sometimes a fresh perspective or an innovative use case is all it takes to turn a product from good to great. However, when it comes to testing, we often find ourselves in unchartered waters wondering if the product will actually work in the diverse customer landscapes. It is virtually impossible to test the vast number of devices and configurations of software that web-based software can run on today. Truly robust testing is time consuming, and ensuring that every possible permutation and combination of features, localizations, and platforms works, as intended is nearly impossible.

Often times, comprehensive testing is a challenge and buggy code is delivered to the customer. For example, if a Software-as-a-Service (SaaS) application does not render in a particular browser or a critical software tool fails to deliver its intended functionality, a bug fix or a patch is promised and the vicious cycle starts all over again. Either way, the customer withstands the worst of inadequate testing, especially when faced with the escalating costs of software maintenance, performance, etc. For the software development company, ramifications include distress around brand image, perceived quality, relationship and potential future projects, trust, etc.

Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards software quality assurance and control. With this new form of software testing, the product is put to test under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all, bug-free.

Crowd sourced testing, conceived around a Testing-as-a-Service (TaaS) framework, helps companies reach out to a community to solve problems and remain innovative. When it comes to testing software applications, crowdsourcing helps companies reduce expenses, reduce time to market and increase resources for testing, manage a wide range of testing projects, test competence needs, exigency to resolve higher defects rates, and use 3rd party’s test environment to subside the project requirements.

It differs from traditional testing methods in that the testing is carried out by a number of different testers from across the globe, and not by locally hired consultants and professionals. In other words, crowd sourced testing is a form of outsourced software testing, a time-consuming activity, to testers around the world, thus enabling small startups to use ad-hoc quality-assurance teams, even though they themselves could not afford traditional quality assurance testing teams.

Why Does Crowd Sourced Testing Work?

To understand why crowd sourced testing works, it is important to understand the set of biases that infest most testers and test managers around the world. This phenomenon is called, “The Curse of Knowledge,” a phrase used in a 1989 paper in The Journal of Political Economy. It means that for a particular subject expert, it is nearly impossible to imagine and look beyond the knowledge the tester has acquired i.e. the set of concepts, beliefs and scenarios that the tester knows or predicts. As a result, it is particularly challenging to think outside the box and conceive the various ways a typical end user would use particular software.

This phenomenon has been empirically proven through an infamous experiment conducted by a Stanford University graduate student of psychology, Elizabeth Newton. She illustrated the phenomenon through a simple game, people were assigned to one of two roles, namely tappers and listeners. Each tapper was to select a well-known song, such as “Happy Birthday,” and tap the rhythm on a table. The listeners were to guess the song from the taps. However, before the listeners guessed the song, tappers were asked to predict the probability that listeners would guess correctly. They predicted 50%. Over the course of the experiment, 120 songs were tapped out, but listeners guessed only three of the songs correctly – a success rate of merely 2.5%

The explanation is as follows: when tappers tap, it is impossible for them to avoid hearing the tune playing along to their taps. Meanwhile, all the listeners could hear is a kind of bizarre Morse code. The problem is that once we know something, we find it impossible to imagine the other party not knowing it.

Extrapolating this experiment to software testing, most testers conduct a battery of tests that they feel is representative and that captures the set of end-user scenarios for how the software would be used. The reality is far from this. Any expert tester would asset that it is impossible to capture the complete set of scenarios that an end user may throw at a software system. As a result, critical path(s) of the code under certain scenarios go untested, which leads to software malfunctioning, production system crashes, customer escalations, long hours of meetings, debugging, etc.

Crowd sourced testing circumvents all these headaches by bringing a comprehensive set of code coverage mechanisms and end user scenarios during the design and development stages of software engineering, during which the cost of modification is meager. This results in identifying critical use cases early on and providing for those contingencies, which reduces software maintenance costs later on during and after productive deployment. Besides progressive code coverage, the quality and depth of software testing among various vital software modules is achieved, which ultimately results in a higher code quality, among other benefits.

Crowd sourced testing – the framework

At the heart of crowd sourced testing is the community that tests a given software product. The community encompasses people from diverse backgrounds, cultures, geographies, languages, all with a diverse approach to software usage. The community, represented by a diverse and extended user space, tests any given software by putting it to use under realistic scenarios, which a tester in the core test team may not be able to envision, given a tester’s constraints, such as limited bounds of operation, knowledge, scenarios. Thus, it is easy to observe the broad set of usage patterns that put the software under intense scrutiny. Crowd sourcing software testing draws its benefits from delegating the task of testing a web or software project, while in development, on to a number of Internet users, to ensure that the software contains no defects.

The method of crowd sourced testing is particularly useful when the software is user-centric, when software’s success and adoption is determined by its user feedback. It is frequently implemented with gaming or mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out internal testing.

The spectrum of issues that such test efforts could uncover within a short lead-time is particularly noteworthy. Such testing efforts yield productive results with reasonable costs. Often times, the product company pays only for those valid reported bugs. Hence, the Return on Investment (ROI) is high compared to the traditional means of software testing.

How does it work?

Most crowd sourced testing companies provide the platform for the testing cycles. Clients specify the type of tests that they wish to have performed and the types of devices that the software product must be tested on.

Testers complete a profile, indicating the skills they have, the devices to which they have access to, and the countries where they reside. Once a tester has completed his profile, he/she can check the project dashboard for a listing of projects and releases that are available for testing. The dashboard may also include sample test scenarios, additional tools and scripts, instructions for testers about what is expected from them, etc. Usually, the testers are required to submit a QA plan, which outlines both high level test cases and detailed test scenarios. The plan may also include whether or not the test can be automated and expected results.

A qualified Project Manager, who is typically a proven community leader or a person from the client/the platform company, reviews such plans, and approves or amends such plans to cater to the client’s specific testing requirements.

Each project includes an explanation and access to a forum where bugs and issues are discussed and additional questions can be asked. Testers document bug reports and are rated based on the quality of their reports. The amount the testers earn increases as their rating increases.

The community combines aspects of collaboration and competition, as members work to finding solutions to the stated problem. Forums facilitate networking and discussion of bugs or relevant issues; rating systems allow for recognition of a job well done, which helps participants gain credibility and improved career.

The crowd source testing team is usually in addition to the organization’s testing team, and not a replacement.

Checks & Balances
Security is a crucial element to crowd source testing. More often than not, confidential customer information is exposed to testers during application testing. Any breach of this data can lead to serious damage, both to the brand and the business. Test data management ensures the availability and security of test data by obfuscating sensitive information for large-scale testing engagements. Masking such information or creating ‘test-only’ data helps maintain privacy and security while using crowd sourced testing services.

In almost all cases, the testers are required to sign a Non-Disclosure Agreement (NDA) when they join the community. The NDA forbids them from talking about customers, their products or specific defects, both offline and online on Facebook, Twitter, personal blogs or anywhere outside the confines of the private testing platform. Beyond that, the customers can upload a customized NDA, which testers must sign before viewing the customer’s project. For projects that require a high level of security, a pre-screened list of white hat engineers, that have a long professional relationship with the platform company are selected.

Comments are off for this post

Importance of Software Testing in the IT Industry

Feb 17 2022 Published by dayat under Uncategorized

Software development companies dedicate a substantial amount of resources and manpower for the development of applications according to requirements specified by enterprises or individuals. However, subsequent to development of such applications/software, companies are required to ensure that such software/applications perform in accordance with the client’s requirements. To ensure that any and all bugs in the software are identified prior to the deployment, various testing procedures are implemented by the IT industry. The importance of this is directly related to the ability of software to measure up to its potential. If the new software is unable to perform the task it was designed for, the client might face severe losses due to stoppage of work and also adversely affect the business of the software development company. The scope of these procedures is to facilitate identification of a problem in the software, fixing of such problems is not within in the scope of software testing. Some of the leading methodologies implemented by companies in the IT industry include General, Load, Functional and Regression Testing.

General Testing

This refers to the general tests carried out on software/applications to ensure the functionality of newly developed software. Some of the common tests included as part of standard Quality Assurance procedures are web performance and usability testing. The web performance testing procedure is mostly engaged in evaluating the online performance of a web-based software application or a website. Usability testing is a mainly subjective approach, which ensures that the software is capable of being utilized effectively in a given set of circumstances. The purpose is to identify the general operating capability of the software/application being tested and to help developers determine some of the areas of improvement for the software. A software development company usually performs general testing of software/applications along with other more advanced methods to ensure that the software conforms to the pre-defined operational capabilities specified by the client/user group.

Load Testing

The load testing procedure simulates operating conditions of the software/application during periods of higher/normal load to gauge the effect of such changes on the functioning of the software/application. This is not the same as stress testing, because load testing checks the operational capabilities in case of both normal load and high load conditions, while stress testing attempts to induce errors in normal operations by using increased system load. This is considered to be a type of non-functional testing, which is undertaken by software development companies to gauge the multi-user support capabilities of the application.

As a commonly employed practice in the software industry, its specific goals are widely disputed and the term is often utilized in conjunctions with volume, reliability, software performance and concurrency testing. By using load testing, developers can attempt to determine the reason for slow performance of software. The common reasons for such slow response commonly include load balancing between multiple servers, client-side processing, network congestion/latency, available database service and/or bugs in the application server(s) or software. The use of load testing is recommended for software/applications, which are subjected to SLA (service level agreement) for ensuring that the software is capable of supporting multiple users. As the procedure simulates an increase in system load by using multiple virtual users, various software are currently available to carry out load testing. Some of the leading load-testing tools used by developers globally are IBM Rational Performance Tester, Apache JMeter, LoadRunner etc. Additionally, a load testing tool commonly favored by software testing companies in India is available as part of the Visual Studio Ultimate Edition of Microsoft.

Functional Testing

This type of testing is a type of black-box testing based on the specifications of the software components being tested. The functions of specific components of the software are feeding inputs and checking the output thus obtained. In functional testing, the internal structure of the program is seldom considered hence, it is classified as a type of black-box testing. The key steps involved in functional testing include identification of functions, which the software is expected to perform, creation of input data according to specifications of the identified functions, determining output based on the specifications of those functions, executing the test scenario followed by comparison of the obtained output vs. the expected output. Functional testing is not the same as system testing as system testing involves validation of a program in comparison to the published system or user requirements, whereas, functional testing is carried out by checking a program with respect to established specifications and available design documents for the software/applications.

Regression Testing

The regression testing refers to any type of software testing, which attempts to identify bugs, which are present in either the functional or the non-functional areas of a system subsequent to making modifications such as configuration and patch changes. The key function of regression testing is to ensure that the use of a patch or upgrade does not lead to the introduction of a new bug into the existing system. Additionally, regression testing helps ensure that the changes in one section of the software do not induce changes in another part of the software’s code. Some of the commonly applied regression testing methods include the use of earlier tests to check for alterations in program operation and the search of any previously fixed bugs, which had re-emerged subsequent to introduction of the new code. Fixed bugs in software often re-emerge and regression testing is one of the leading methods to ensure that such re-emergence is identified and easily controlled before any lasting damage occurs. Software development companies repeatedly perform regression testing of software/applications after any change in coding such as use of patches etc. to ensure that the functionality of the application is unimpaired. Such repetitive testing is usually automated by using an external tool such as Bamboo, TeamCity, Jenkins, Hudson, Tinderbox or BuildBot. This type of testing is generally performed by the QA team in case of leading software development companies, however, smaller companies are often engaged in outsourcing such services to companies specializing in the field of software QA and testing.

What’s Next?

As new technologies emerge, more testing procedures are being developed and implemented by organizations all over the world to ensure that new software perform according to their requirements and specifications even when stress or when additional functionality is introduced into the software. The emerging testing solutions, which are powered by new technology, are designed to reduce the time and resources required for testing in order to streamline the quality control / quality assurance services associated with software development. Some additional types of testing, which are currently used in the software industry are white box testing, system testing, non-functional testing, acceptance testing and integration testing. Each of these testing was developed to identify and resolve application/software limitations in a specific set of conditions; hence they are useful for software testing carried out in case of specific quality assurance and testing procedures.

Comments are off for this post