By Ran Rachlin, Ubertesters
If you don’t like it, wait a minute and it will change.”
That is what is said about the weather in…pick a place. It’s been said about New England, Louisiana, Oklahoma, London, and even Paris. Probably everywhere.
It’s also a reasonable statement to make about developing apps. There’s no app for what you want? Wait a minute, or maybe two, and there probably will be.
The rapid pace with which apps arrive at the user marketplace suggests that they are developed in haste, and that it will be the users who sign up for them who will repent their registration at leisure.
This is not the case. At least, not with reputable, reliable app developers.
Rather, they submit their applications for rigorous beta testing. Consider beta testing to be the pre-release testing, sometimes conducted by beta users, who use the app regularly and provide feedback, and sometimes conducted by professional QA analysts, who test functionality, usability, regression testing, and all manner of rigorous evaluation that one would expect from a professional QA team. By the time the app hits the marketplace, even buried bugs or kinks should have been discovered and removed or resolved. The general consumer should experience flawless performance – and in fact must do so, as today’s consumers have zero tolerance to bugs or crashing applications, and would immediately delete the app in the event that it worked poorly.
Who are These Beta Testers?
Developers have four different options when it comes to finding professional and experienced QA people to test their apps, and each option has its advantages and disadvantages, as far as the developer is concerned. Specifically, they make use of any one of the following four ways of doing beta testing:
1. In-house testers
The obvious advantage of in-house testers is that they are local, available whenever you need them, and are likely to know the product, or at least many aspects of it. The challenge of in-house testers is that they are likely to be few in number, and the QA team often must round out its numbers to increase the testing by using people from the company’s IT department, the developers, and so on. These folks have other jobs they are supposed to be doing, so the beta testing therefore is given lower priority, or handled as top-priority by people who may well want to be doing other things. Alternatively, those from the development team may be too close to the product to carry out unbiased testing. Developers can also hire additional in-house testers, however the hiring and on-boarding process is quite time-consuming, slowing productivity and efficiency. Additionally, as the testing cycle is cyclical, there will be times when a developer is paying the in-house testing team for essentially doing no work, which obviously cuts into profits.
2. Outsourcing
When beta testing is outsourced to professionals, a company should be able to rest assured that the testing will be done thoroughly and well. The challenge of outsourcing is that too few devices with the app reach too few testers for rigorous, thorough testing to be achieved. One room of several dozen testers is simply insufficient to put the app through its paces. Also, outsourcing testing incurs additional expense. Additionally, to successfully outsource beta-testing, the developer will have to hire different outsourcing companies in each of the markets he plans to make the app available, which becomes very difficult to manage. History has demonstrated time and again that very few companies have the means to successfully manage more than one outsourced vendor at a time.
3. Marketplaces
Testing in the marketplace means making sure that you have lined-up the appropriate professional QA team to test your product and that you are able to manage them effectively. This is particularly difficult to ensure, because until the testers have a track record with your software, you cannot be confident that when they don’t find bugs, it’s because the bugs are not there to be found (perhaps the testers simply aren’t good at finding them!).It is impossible to know up-front if the testers are good or not – reading CVs and getting references is good, but it is far from ensuring that you will have the rightteam for your app in place. Additionally, the very fact that you must manage an unknown team decreases the feasibility of this kind of tester. This is probably the worst option if you need more than 2-3 testers.
4. Crowdsourcing
Even if crowdsourcing weren’t all the rage for everything these days, there are a lot of reasons why it’s a sound way of beta testing, including its cost-effectiveness (companies tend to pay the testers a low hourly rate or pay only when they discover valid bugs). The populations that agree to do the beta testing are likely to be diverse (different devices, different computing environments, different operating systems for example), extensive, and relatively unbiased, with potential experts spread out geographically. You can also recruit specific target groups without having to depend on your potential customer base. Those who do test via the crowd need to be mindful of issues of confidentiality, of course, and realize that communication with the group may be difficult or delayed. A crowdsourcing company, however, can ease some of the stress of the testing process by assuming the burden of managing the testing process. This method also requires excellent oversight, as the population is diverse, so when a crowdsourcing company provides one point of contact, it has its definite advantages. Finally, make sure the crowdsourcing provider offers the right tools to truly monitor and manage the testers’ activity and results, otherwise it will be difficult to assess the effectiveness.
What Do Beta Testers DO?
No matter the product, it is the responsibility of beta testers – including both beta users and QA teams – to use the app as naturally as they can – and as thoroughly as they can. That is, they are tasked with exploring every feature and function of the app as if they would use every feature and function of the app. This kind of rigorous handling is particularly effective at finding bugs, including in the particularly tricky area of interoperability.
Beta testers provide all of us who rely on our mobile apps for more and more these days with an invaluable service. In rooting out the bugs, they make our user experience more enjoyable, and – critically – more reliable. We owe them a debt of gratitude.
Ran Rachlin is the CEO and Co-Founder of Ubertesters, a provider of crowd-based, process management tools for mobile applications testing.
Comments