Remote unmoderated usability testing — pros and cons

unmoderated user testing

Unmoderated user testing: making user testing cheap, fast and efficient

Ever wondered why unmoderated user testing is so popular in the UX community? What even is this type of testing and how is it beneficial? In this article we will try to answer all the questions you may have — and maybe provide some additional information on the top of it. We will discuss the undeniable benefits of unmoderated user testing as well as the tricky parts. Last but not least, we will analyze the process of setting up an unmoderated usability test and hopefully, at the end of the article, you will understand people’s obsession with this type of usability testing.

What is unmoderated user testing?

The main difference between moderated and unmoderated user testing is that for unmoderated testing, you don’t need a trained person to be there during the study — actually you don’t need anybody to be there. All you have to do is to set up the study and get people to complete it. This also means that for a moderated study there can only be one participant (or a very small group of participants) at a time. For an unmoderated study, on the contrary, the number of participants is unlimited. Because participants complete the study by themselves, no supervisor is needed. As a result, with unmoderated testing, you are able to collect much more information in a much shorter time.

Pros and cons of unmoderated user testing

PROS

Less time consuming

Less expensive

Limitless respondents

Fast feedback

CONS

Setup

If participants struggle or get confused whilst completing a task, you won’t be able to guide them or help them in any way. In such a situation the participants are very likely not to finish the task. It is thus crucial to set up the study properly.

Less details

How to set up a good unmoderated study using UXtweak

Prepare tasks and detailed instructions

Firstly, not being in a UX lab, it will be harder for people to focus for a long time and it’s more likely they won’t finish the study. That is why the tasks shouldn’t take too long to complete. Since the time for individual tasks should be kept short, unmoderated user testing is rarely used to test the usability of the whole product. This means you have to set up shorter condensed tasks in order for people to successfully complete them.
Doing more studies for shorter tasks doesn’t necessarily have to be a disadvantage, as it permits you to collect more detailed information. Conducting multiple types of studies is not a problem with UXtweak as it offers a wide range of features. You can use a different feature for every aspect you want to study. Carving up all the things you want to know into smaller tasks, you can for example use tree testing to evaluate an existing information architecture or card sorting to create intuitive content structures to your website visitors.

Secondly, you need to describe the tasks as precisely as possible. Remember, there won’t be anyone to help the participants in case they struggle. This means that the instructions have to be as exact as possible. Imagine someone who doesn’t know anything about the task and try to write the instructions for such a person. Fortunately, UXtweak allows you to precisely inform your participants about the study. Apart from written instructions, you can also illustrate the instructions by including instructional graphics.

In order to get the best information possible, it might be a good idea to use UXtweak’s option to ask followup questions. Asking specific questions about individual parts of the task will deepen your understanding of the participant’s behavior. These questions can be multiple choice questions or open-ended questions, depending on the information you want to obtain. They can be asked after each individual task as well as after completing the study.

Recruiting respondents

At UXtweak we believe that the best respondents are respondents that are as close to your real customers as possible. That’s why we don’t use a pool of respondents. Because these people do such studies often, they know what aspects to focus on in order to complete the task as soon as possible. Thus their behavior isn’t completely natural and they don’t represent your real customers.

That’s why we developed two ways to recruit respondents as similar to your real customers as possible.

The first option is to post a link to the study on your social media or to send it to your potential respondents by email or through CRM. All the respondents have to do is to click on the link and they will be redirected to the study. Once the study opens in their browser, they are all set to start.

The second option is to use UXtweak’s recruiter widget. There is no way to get respondents more similar to your customers than by inserting a recruiter widget to your website — which is really easy to do. Using this method, you are able to turn visitors of your website into respondents. If you want to be totally sure that they will participate, you can motivate them by a reward for completing the study (e.g. coupons).

Analyzing the results

After you end the study, the first thing you see is a dashboard. This dashboard contains the most important information about your respondents (number of respondents and respondents that successfully completed the study), time taken to complete the study and the most recurring geographical locations of your respondents.

You can analyze your respondents (or visitors in the case of a RePlay Visitors study) in more detail. For every respondent, there are details such as his or her city, region and country, operating system and screen resolution. You can also see for every individual participant if he or she completed the study and how much time it has taken for him or her to complete it.

The next thing to analyze are the study results themselves. This analysis varies according to the type of study. Insights from a tree testing study include, among other things, a detailed analysis of your participants’ path or an analysis of which branches were clicked first, while the information obtained from a card sorting study include for example the results matrix (the number of times each card was sorted into your pre-set categories) or the popular placement matrix (what he most popular placement for each card was). What these analyses have in common is that they clearly explain what happened during the study and help you to understand what the problem of your website is.

In order to make it even easier to interpret the study results, UXtweak offers various graphs, diagrams as well as heatmaps and scroll maps. These graphic representations of the collected data make the analysis of quantitative data extremely intuitive and easy to understand.

The final step you may have to take is presenting these data to your stakeholders. UXtweak offers you a very simple way to do so — all you have to do is to send them a link to the results of your study. A very nice feature available with card sorting studies is the option to make a PDF report of the study, making it even more attractive to your superiors or stakeholders.

Summary

--

--

UXtweak is unique set of research tools to conduct qualitative studies on your website #UXResearch #UXTesting

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
UXtweak

UXtweak is unique set of research tools to conduct qualitative studies on your website #UXResearch #UXTesting