I really don’t like to go grocery shopping. There are a lot of things I don’t like about it, but especially trying to choose the “right” product. There are just too many choices. Do we really need this many types of mustard? Unmoderated usability testing tools used to fit nicely into the old general store. There were just a few tools, and the differences were obvious. Recently, unmoderated usability tools have begun to fill an entire American sized grocery store. In this article I would like to help you walk down some of the aisles of this store, and provide a little guidance in how to shop for the right unmoderated usability testing tool. Think of me as your “personal shopper”!

Important note: In the first version of this article we’ve mistakenly shared wrong information regarding the services that Webnographer provides. Webnograper is a service that carries out studies that range from 100 to over a 1.000 people, resulting in both qualitative and quantitative research data. It provides many different services.

Before we enter the grocery store, there are few ground rules I want to lay down. First, we are only shopping for unmoderated usability testing tools. This does not include remote moderated testing tools such as Morae, WebEX, or GoToMeeting that can used to run 1:1 usability sessions. We are also not shopping for online survey tools such as SurveyMonkey and SurveyGizmo. These are certainly helpful in collecting feedback from large numbers of participants, but there is typically no interaction with a product or design. We will be focusing on tools that researchers and designers can use to collect usability data from end users in an unmoderated way. These tools essentially involve setting up a study, launching it to a small or large number of participants, and then some conclusions might be drawn regarding the usability of a specific product or design. Before we enter the store, I apologize in advance.

There are some tools that I will miss or may misrepresent. The tools are changing everyday in terms of functionality and pricing structure. This is only a fuzzy snapshot in time. With this in mind, now let’s enter the store. As you can see from the “floor plan” below, there are two main sections of the store – quantitative and qualitative-based tools. Quantitative-based tools are designed to collect data from a large number of participants, with a focus on UX metrics such as task success, task completion times, click paths, abandonment rate, etc. These are the tools you need to run a benchmark usability study or compare subtle design treatments. The other side of the store is the qualitative-based tools. These tools are essentially a substitute for lab or remote-based usability testing. These tools emphasize collecting feedback from a small number of end users in a quick and dirty fashion. You may or may not be able to actually derive any metrics, but at the very least you will gain some insight into the most significant usability issues and hopefully make the right design decisions.

Quantitative Tools:

Full-Service Tools

There are only two full-service tools I am aware of – Keynote’s WebEffective and Imperium’s Relevant View. Both of these tools provide you the flexibility you need in terms of setting up an online usability test and the data it can provide. You will have a tremendous flexibility in the study design including the use of conditional logic, blocking, and randomization.  But, what sets these two tools apart from the other quantitative options is the support you receive. Experienced researchers will assist in designing the study, piloting/launching, and data analysis. Of course, this support does not come cheap; an individual study can easily be $10K and up. There is no set price; it all depends on the individual characteristics of your study and how much help you need. But, you will get peace of mind that the study was conducted in a highly professional way.  You can just think of this as the “prepared food” section of the store.

Self-Service Tools

These three self-service tools range in the degree of flexibility and analytical capabilities. UserZoom has many of the same set of functionality as the full-service options, but at a fraction of the cost. A typical usability test is about $3K, or $9K for an annual license. Loop11 is much more streamlined, in that it does not support functionality such as conditional logic or randomization. Loop11 is easy to use, very inexpensive ($350 per study), but it is very basic. If you need to collect various UX metrics, you want to visualize abandonment or click paths, or really do anything beyond tasks and simple questions, you should consider UserZoom. Bottom line though, and of these three tools will help you collect solid usability metrics at a reasonable cost. They also offer technical support as well as access to customer panels.

Card-Sorting/IA Tools

These online tools allow researchers to create or validate the information architecture for any website. The folks from Optimal Workshop have created two valuable tools. One tool, OptimalSort, is a cardsorting application, in that it allows participants to sort items into groups, and provide labels for those groups. It then analyzes the data from all the participants, and determines the best groupings for the items. This is sometimes referred to as an open card sort. The Treejack application allows the researcher to validate an information architecture. Participants are given a set of tasks, and then asked to select the correct location for each task. Task success and time are measured as a way to evaluate and validate the IA. This is sometimes referred to as a closed card sort. WebSort offers very similar functionality as OptimalSort. The pricing for all three products is very reasonable, with single studies running about $100.

“Do-it-Yourself” Tools

For researchers on a very tight budget, or who have very simple needs might want to consider a DIY approach. All the details can be found in our book, Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies (Morgan Kauffman/Elsevier, 2010). The basic idea is that the researcher will use a little bit of java script and HTML to create a simple web page that introduces the study and launches two separate browser windows: One browser will contain the website being evaluated, and the other browser will contain the survey itself (usually very small, above or below the website browser). The participant will move between each browser as they go through the study. While data are not passed across the two browsers, you can still capture basic data such as task success, completion times, and self-reported metrics.

Qualitative Tools:

Video Tools

Video tools are roughly equivalent to having a lab study on autopilot. They are quick and easy to set up, and are inexpensive. The researcher simply provides a set of tasks and a URL, and identifies the targeted participants. Participants are automatically recruited who fit the profile. The participant is carrying out the tasks, while their behaviors and verbal comments are being captured by a webcam. Once the study is finished, the researcher can view a video of all the sessions and distill some of the high level issues. There is little, if any opportunity to collect any data from these sessions. However, they may work well for the researcher who needs a quick sanity check of a design. The cost is very reasonable, usually less than $20 per participant.

Report Tools

These tools provide researchers with a detailed report from a user session.  Similar to the video tools, the researcher sets up a study by identifying tasks, but they also include a set of open ended questions that the participants answer as they interact with the website. Participants are recruited based on select criteria and participate in the study. The output is a detailed report of each answer by each participant. This is a very easy way to get some high level feedback on a design. While it might take some work to identify the significant usability issues, it is very easy to use the quotes to compliment an in-person study. The cost is also very reasonable, typically less than $20 per participant.

Click and Mouse Tools

Click and mouse tools provide the researcher click and mouse movement data on a website. Unlike the other tools that capture comments (video or written), these tools focus on how users move throughout a website.  There are some very nice visualizations that come out of these tools, such as click maps, attention maps, mouse movements, and scroll maps. These tools are particularly effective at quickly capturing data about what is drawing user’s attention. For example, fivesecondtest.com is very simple in that you can show static images of web pages to participants for five second and participants click on those features that initially grab their attention. Of these tools, ClickTale seems to offer the most functionality, including metrics such as abandonment rates as well as video recordings. The cost for these tools is very reasonable, usually less than $20 per participant or a monthly license for a few hundred dollars or less.

Combination Tools

Combination tools collect a lot of data about the user experience. Some of the data are qualitative such as videos of the sessions and verbatims. Other data are quantitative such as clicks, task success, keystrokes, pages visited, and completion times.  Unlike the self-service tools, these tools are typically used for smaller sample sizes (n<20). They provide a nice solution to the researcher who wants to gain a more complete picture of the user experience with a small number of participants. Pricing is quite affordable, typically a few hundred dollars or less for a single study.

I love the TV show Myth Busters because it challenges what I think is true. In the show, popularly held myths are tested in an entertaining and somewhat scientific way. My favorite part of the show, other than the explosions of course, is when my beliefs turn out not to be true. This always keeps me open minded, and focused on reality. I also enjoy being on the other end – exposing myths as unfounded. This is the perspective I am taking for this article. Specifically, I will be highlighting five common myths related to online (unmoderated) usability testing, and explaining why I think these myths are not true.

The motivation for this article is to help UX researchers keep an open mind about online usability testing. There are some researchers who have been using this approach for years and find it useful (in certain situations). Others are new to it, and wanting to learn more about its strength and limitations. Finally, some UX researchers have already formed an opinion about online usability testing, and deemed it not useful for a variety of (unfounded) reasons. I hope by exposing these myths, we (as a UX community) can evaluate this tool based on its actual merits.

1.    There is too much noise in the data to be trustworthy

Perhaps the most common myth about online usability testing is that the data are not very reliable. Some people will say that the participants don’t try very hard, get distracted in the middle of the study, go through the study as fast as they can to get the incentive, or even misrepresent themselves to qualify for the study (in order to get the incentive). This is certainly true, and generally occurs in 5% – 10% of all cases. But, there is good news!

There are some very useful techniques to clean up the data. Identifying (and removing) “mental cheaters” is not very hard. These folks behave in highly predictable ways. “Flat lining” is when someone answers all 1’s or 5’s on a 5-point rating scale.  “Speed traps” are one way to identify these folks (see question 6 in the figure below) by checking to make sure they are actually reading each question. Consistency checks can be used by re-wording questions, and analyzing the consistency of responses. Removing unrealistic task times is also very easy to do, and can help deal with the participants who went home for the weekend in the middle of a task. Also, screening questions can be written to minimize the number of fraudulent participants (those that misrepresent themselves) who take part in the study.

Bottom line, the data from an online usability test can be just as reliable as a traditional lab test. The only difference is that a little cleaning up needs to happen before jumping into the analysis phase.

2.    You can only collect quantitative data

I am always surprised when people tell me that online usability testing is only useful for collecting basic metrics such as task success, completion times, or satisfaction. For some reason, people assume that just because you have a large sample size, you can only collect quantitative data. Online usability studies just don’t make sense if you want to collect qualitative data about the user experience. This cannot be farther from the truth.

One of the beauties of online usability testing is that the researchers can collect a rich set of both quantitative and qualitative data about the overall user experience. For example, if a user has difficulty with a specific task, they can provide comments as to why they had difficulty with that task, or what they were expecting to happen at different points along a transaction. Qualitative data, usually in the form of verbatim comments, can be collected at any point in their experience. The user can even be prompted to provide verbatim comments when they exhibit certain behavior, such as abandoning a transaction or deviating away a desired navigational behavior.

Verbatim comments are not only easy to collect, but they are becoming much easier to analyze. There are many tools available now to pick out patterns in verbatim responses. Word clouds (see below) are one example to way to get a quick sense of the key patterns in verbatim responses.

3.    Online usability testing does not work well during the design phase

Most people think of running an online usability study only after the product has been built. It only makes sense to run an online usability study as part of a summative evaluation, and doesn’t really fit in during the actual design phase. Online usability testing can’t inform the design, takes too much time, or is too expensive to conduct during the design phase. These are all untrue.

Online usability studies can be set up within a few hours, and data collected in a matter of hours. We have set up, launched, and analyzed data from an online study within the same day. Not only can a study be set up quickly, but it can help answer questions that typically come up in the design phase. For example, if there is a question about preferences around navigation method, labeling, or look and feel. These and many other design preference questions cannot be reliably answered with a small sample size. Sometimes, we need to gather data quickly from hundreds or thousands of users in order to validate significant design decisions.

4.    It only works with websites

One of the things that researchers quickly point out is that online usability testing can only be used for evaluating websites. It simply doesn’t work for software, voice response systems, mobile, consumer electronics, and toaster ovens. While a vast majority of online usability testing does focus on websites, it doesn’t have to.

Over the years, we have used an online usability testing approach to evaluate non-web interfaces. Conceptually, it is still the same method. The participant is asked some questions, given some tasks, and provides feedback about their actual experience. The only real difference is that the participant’s behavior is not being tracked. But, a lot of useful data can be collected about their experience, such as whether they were successful in completing their tasks, how long it took them, what they felt about their experience, and where they had difficulty. So, even though you might be giving up a little behavioral data, there is still a lot to be learned about their experience.

5.    Online usability testing costs too much

Some people say that they would love to do more online usability testing but it is simply too expensive to use on a regular basis. It certainly can be pricy, but fortunately, in the last few years there are a variety of tools that allow you to run online usability studies for not a lot of money. There are various self-service providers to allow you to set up and run your own online usability study. You only need to pay to access their technology. Visit www.measuringux.com to see a complete listing of vendors.

If budgets are really tight, there is a way to run your own “discounted” flavor of an online usability testing for free, or practically free. By taking advantage of some online survey tools (such as Survey Gizmo or SurveyMonkey), and a little html and java script, you can literally create your own online usability study for close to free (see the figure below as an example of the “homegrown” approach). While you will give up a little data and functionality, it can be useful in those situations when you have no budget to run an online usability study. Visit www.measuringux.com to see more details about how to create a discounted online usability study.

Try it for yourself

I may or may not have convinced you that these myths are untrue. Regardless, I would encourage you to consider online usability testing as part of your user experience design and research efforts. In doing so, you will discover its’ strengths and limitations. After all, every user research method has its own strengths and limitations. Online usability testing is no exception.

Editorial note: Interested in learning more about this subject? Bill recently co-authored Beyond the Usability Lab: Conducting Large-Scale Online User Experience Studies.

Top image: Brad Montgomery / cc