Remote unmoderated usability testing — pros and cons

Unmoderated user testing: making user testing cheap, fast and efficient

Interested in a type of user testing that runs (almost) by itself, is efficient and can provide you and your team with valuable feedback on your website or prototype? You are in the right place.

Ever wondered why unmoderated user testing is so popular in the UX community? What even is this type of testing and how is it beneficial? In this article we will try to answer all the questions you may have — and maybe provide some additional information on the top of it. We will discuss the undeniable benefits of unmoderated user testing as well as the tricky parts. Last but not least, we will analyze the process of setting up an unmoderated usability test and hopefully, at the end of the article, you will understand people’s obsession with this type of usability testing.

What is unmoderated user testing?

Unmoderated user testing is a type of usability testing, where respondents are not supervised. The study is prepared in advance and the respondents complete it by themselves, without any external influence. This type of testing is a great way to test specific parts of a product and to collect valuable feedback on websites. It is frequently used by UX teams all around the world as it is useful throughout a product’s whole lifecycle and gives UX researchers and designers a lot of insights into user’s behavior and thinking.

The main difference between moderated and unmoderated user testing is that for unmoderated testing, you don’t need a trained person to be there during the study — actually you don’t need anybody to be there. All you have to do is to set up the study and get people to complete it. This also means that for a moderated study there can only be one participant (or a very small group of participants) at a time. For an unmoderated study, on the contrary, the number of participants is unlimited. Because participants complete the study by themselves, no supervisor is needed. As a result, with unmoderated testing, you are able to collect much more information in a much shorter time.

Pros and cons of unmoderated user testing

Both types of user testing — moderated and unmoderated — have their benefits as well as drawbacks. Being the opposite of moderated user testing in many aspects, unmoderated studies have certain benefits that are unique to this type of usability testing. It does have a few downsides though, which have to be remembered when deciding which user testing to go for.

PROS

Less time consuming

Firstly, unmoderated studies are much less time consuming to conduct. Although you will probably spend more time preparing an unmoderated study than a moderated one, you will still save all the time you would spend in a UX lab or online moderating the study for every respondent separately.

Less expensive

Secondly, you will save a LOT of money. When conducting a moderated study you have to pay the researcher conducting the study and you also have to pay the respondents for their time. These things can add up to quite a big amount. With unmoderated studies, you pay for usability testing tools and that’s all. You can of course motivate your respondents, but such motivation comes mostly in the form of coupons. Check out the pricing for UXtweak.

Limitless respondents

Thirdly, your potential respondent range is basically limitless. Thanks to the internet, recruiting a respondent on a different continent takes just as much effort as recruiting someone from your street. This gives you the opportunity to find respondents fitting any profile you need.

Fast feedback

Last but not least, with unmoderated studies you can obtain more data quicker than with moderated studies. In most cases, to recruit respondents for unmoderated studies either a link is sent to your audience or you are provided with respondents from a pool of testers of the tool you use. As soon as they receive the link to the study, they can start completing the study and you start collecting large amounts of data within a few hours.

CONS

Setup

Setting up an unmoderated study is a very specific process. Because it is different to setting up a moderated study, you have to keep on mind all its particularities. Due to the absence of a person conducting the study, it is not possible to react or interact with your participants. You have to know exactly what you want to find out with the study and thus what questions to ask, how to instruct the participants etc.

If participants struggle or get confused whilst completing a task, you won’t be able to guide them or help them in any way. In such a situation the participants are very likely not to finish the task. It is thus crucial to set up the study properly.

Less details

Most of the time when conducting an unmoderated study, you don’t see your participants. This means you can’t evaluate body language, facial expressions or behavior details, meaning you won’t be able to obtain all the possible data.

How to set up a good unmoderated study using UXtweak

Prepare tasks and detailed instructions

When setting up an unmoderated study, you have to keep in mind its limitations. Remember, that people will be at home and without any external guidance. This means two things.

Firstly, not being in a UX lab, it will be harder for people to focus for a long time and it’s more likely they won’t finish the study. That is why the tasks shouldn’t take too long to complete. Since the time for individual tasks should be kept short, unmoderated user testing is rarely used to test the usability of the whole product. This means you have to set up shorter condensed tasks in order for people to successfully complete them.
Doing more studies for shorter tasks doesn’t necessarily have to be a disadvantage, as it permits you to collect more detailed information. Conducting multiple types of studies is not a problem with UXtweak as it offers a wide range of features. You can use a different feature for every aspect you want to study. Carving up all the things you want to know into smaller tasks, you can for example use tree testing to evaluate an existing information architecture or card sorting to create intuitive content structures to your website visitors.

Secondly, you need to describe the tasks as precisely as possible. Remember, there won’t be anyone to help the participants in case they struggle. This means that the instructions have to be as exact as possible. Imagine someone who doesn’t know anything about the task and try to write the instructions for such a person. Fortunately, UXtweak allows you to precisely inform your participants about the study. Apart from written instructions, you can also illustrate the instructions by including instructional graphics.

In order to get the best information possible, it might be a good idea to use UXtweak’s option to ask followup questions. Asking specific questions about individual parts of the task will deepen your understanding of the participant’s behavior. These questions can be multiple choice questions or open-ended questions, depending on the information you want to obtain. They can be asked after each individual task as well as after completing the study.

Recruiting respondents

Alright, you know how to set up a study. Now you need people to participate in it. Before you recruit some people, think about what your target group looks like. Consider things like gender, age, geographical location or the respondents’ knowledge of your product or service.

At UXtweak we believe that the best respondents are respondents that are as close to your real customers as possible. That’s why we don’t use a pool of respondents. Because these people do such studies often, they know what aspects to focus on in order to complete the task as soon as possible. Thus their behavior isn’t completely natural and they don’t represent your real customers.

That’s why we developed two ways to recruit respondents as similar to your real customers as possible.

The first option is to post a link to the study on your social media or to send it to your potential respondents by email or through CRM. All the respondents have to do is to click on the link and they will be redirected to the study. Once the study opens in their browser, they are all set to start.

The second option is to use UXtweak’s recruiter widget. There is no way to get respondents more similar to your customers than by inserting a recruiter widget to your website — which is really easy to do. Using this method, you are able to turn visitors of your website into respondents. If you want to be totally sure that they will participate, you can motivate them by a reward for completing the study (e.g. coupons).

Analyzing the results

At last, you successfully conducted the study and obtained a lot of data. But these data still need to be analyzed in order to be useful. Luckily, UXtweak offers you multiple ways of analyzing data from your study and enables you to share them with others.

After you end the study, the first thing you see is a dashboard. This dashboard contains the most important information about your respondents (number of respondents and respondents that successfully completed the study), time taken to complete the study and the most recurring geographical locations of your respondents.

You can analyze your respondents (or visitors in the case of a RePlay Visitors study) in more detail. For every respondent, there are details such as his or her city, region and country, operating system and screen resolution. You can also see for every individual participant if he or she completed the study and how much time it has taken for him or her to complete it.

The next thing to analyze are the study results themselves. This analysis varies according to the type of study. Insights from a tree testing study include, among other things, a detailed analysis of your participants’ path or an analysis of which branches were clicked first, while the information obtained from a card sorting study include for example the results matrix (the number of times each card was sorted into your pre-set categories) or the popular placement matrix (what he most popular placement for each card was). What these analyses have in common is that they clearly explain what happened during the study and help you to understand what the problem of your website is.

In order to make it even easier to interpret the study results, UXtweak offers various graphs, diagrams as well as heatmaps and scroll maps. These graphic representations of the collected data make the analysis of quantitative data extremely intuitive and easy to understand.

The final step you may have to take is presenting these data to your stakeholders. UXtweak offers you a very simple way to do so — all you have to do is to send them a link to the results of your study. A very nice feature available with card sorting studies is the option to make a PDF report of the study, making it even more attractive to your superiors or stakeholders.

Summary

Unmoderated studies are an easy and fast way to get a lot of very useful information about how your users behave on the website. With UXtweak, you have everything you need to conduct various usability studies. Recruiting respondents that fit your needs is easy — just send them a link or use the recruiter widget. Your respondents do not have to install anything, after clicking on the link they are set to go. Even though direct interaction is not possible, you can ask the respondents questions about their experience after the study or after each task. All results are analysed right after you end the study, providing detailed information about every aspect of the study. This allows you to understand the problems on your website and to fix them at a very friendly price. All you have to do is sign up at Utweak.com!