Many of our clients rely on unmoderated research tools such as usertesting.com, hotjar.com, and so on. We’ve always believed that any research is better than no research. But our clients are often surprised when we tell them about some of the pitfalls of these tools. They are surprised to hear why a session moderated by an experienced researcher is much more insightful.
Are you getting the full picture? Probably not, so, here are our top insights into why unmoderated testing possibly isn’t giving you the results you need.
In case you’re not sure, a moderated session is one where a trained user researcher sits alongside a participant, either physically, or remotely through a video chat and screen-sharing tools. The moderator can listen, probe, and react to what the participant says and does. An unmoderated session is where the participant takes part in pre-determined tasks on their own.
Read on for seven considerations when choosing between unmoderated and moderated user testing.
1. The importance of designing tasks is overlooked
A typical unmoderated task might be something specific like this:
Can you find a Greenland Parka and add it to your wishlist.
The issue here is that there’s a big assumption that users would know the name of this product and search by name. They are very likely to go route one, straight to the search, and this may not tell you much.
In a moderated session, a researcher would be able to show the participant a picture of the product and ask them to find it, giving a more authentic portrayal of how they would navigate or search for a specific product without being lead there by a product name or category.
Sign up to the newsletter
In this task, users may find the option to subscribe and complete the task successfully. Imagine if everyone completes this task with 100% success rate, you would assume that there’s no issue with the newsletter. But what if your real research question was, why aren’t people signing up to the newsletter. Your task tested if people can do it, not if they would want to. This kind of closed task can’t tell you that.
Using moderated testing, we would allow the participant to interact with the site naturally and complete typical tasks as they usually would. We would ask what they would typically do if they wanted to get updates from the company about new products or offers and get them to show us how. We would then probe about which channels they use the most and why. If newsletter wasn’t their choice, we would probe further to find out why and what the site would need to do to offer them something of value to subscribing.
2. You need time and skills in interpreting qualitative research
Often overlooked is the time cost associated with these tools. They rely on you to review all the raw data and watch all the videos. There is some skill in interpreting the findings too. Without a team of experienced UX people to support you, you can get lost in more questions about your results, or find that you are making some big decisions based on one user.
3. You are likely too close to see the real causes
You are so close to your product, and when you are too close, it becomes tough to spot the real issue users are having. It’s so easy to fill in the gaps and jump to conclusions about what you’re seeing. You know how the entire process works; you know why things are the way they, and this can cloud your thinking and prevent you from seeing the best opportunities for change.
Experienced, independent researchers lack intimate knowledge by design. Clients often want to tell us everything about their product or service at the start, but often we like to hold back a little and get just enough information but not too much. Otherwise, we’ll see what they see and lose our independent, objective perspective. Our researchers are explicitly trained only to take notice of what users are doing, not so much what they are saying. This allows us to come up with solutions that our clients have never thought of before.
4. Beware of the career testers
Most of the participants taking part in your tests in unmoderated platforms will be seasoned testers who are experienced in reviewing websites. They take part in multiple sessions per week and know how to play the system to get paid. There’s a tendency for them to offer feedback that they expect you will want to hear rather than how they would use something. In addition to this, with some of the more popular unmoderated services, there is no way to specify the type of user you want to take part in your research, without paying a huge fee. There’s a solid possibility that you are not seeing real people that would genuinely use your product or service.
Whenever we conduct moderated research, we work closely with our clients to identify the exact target audience we need to recruit. We carefully design a screener questionnaire to prevent people from knowing what to say to get selected and get money for an easy job.
5. Technical issues or hiccups can go unnoticed until the end
A common issue we see clients run into with unmoderated research is that they notice a problem with the test when reviewing the videos. By then it’s too late, and your study is already over and paid for. All participants can repeat something as simple as a broken link, or an overly leading question before noticing it.
We always carefully design our research and then run pilot tests to refine them before we start. But the main advantage of moderated research is that it allows you to run a session, adapt and refine it either during, or after each session. This flexibility gives you the best value from each participant.
6. Unmoderated research doesn’t allow you to probe further
Some participants can be quiet or a little mumbly, and some aren’t good at ‘thinking aloud’ which can lead to a frustrating playback experience with unmoderated research. It’s hard to understand or even hear what they are saying. If they are not saying anything at all, the value of that session is lost or left to assumption. Sometimes a participant can even say something that’s really interesting, and without a research moderator being there to pick up on it and probe further, any gems of insight are lost.
With moderated research, the researcher is trained to encourage participants to explain what they are thinking. They know when to sit back and be quiet and when to probe or encourage them to explain why they are hesitating. This is an essential part of the research process and often leads to the biggest Aha moments.
7. Bringing the research to life and winning hearts and minds can be tough
After you have spent a vast amount of time working through your findings, you then need to share your results with the wider business. It can be a hard thing to do when you don’t have experience in presenting research findings. It can be hard to bring the human user to life.
With moderated testing, we offer clients the ability to watch the sessions live via a video feed either remotely, or in person at our research facility. When we get clients to come along and watch the research as it happens, where teams are all sat in the same room together, watching real people you can see the real issues and work together on solutions you all believe in. You are also able to spot those magic moments where a 5-second clip could be generated with will deliver so much more impactful than a 30-minute presentation would.
When working with us on a moderated research project, we will do all the hard work for you by shaping our report to focus on the key areas. We’ll use our experience to convince your bosses and the wider business of the value of the research insights and help them understand the problems and what needs to be in the roadmap to make a real difference to the user experience.
If you’d like to find out more about moderated research and how we can support you in delivering actionable insights from user research, please get in touch. One of our experienced researchers would be happy to talk you through the process and demo our approach: contact us
