Online User Testing | 5 reasons to avoid it

keep out
Online user testing solutions like have become incredibly popular in the last couple of years; providing a fast and easy-to-set-up test platform it’s no surprise it’s become a regularly used tool for CRO agencies, international web enterprises and thousands of small companies.

With plenty of reviews available online there’s no real need for me to go through the list of advantages, however as a strong believer in low cost usability testing proposed by Jacob Nielsen and brilliantly promoted by Steve Krug I think it might be interesting to have a bit more of a sceptical look at these solutions. Mostly importantly I want to try to comparing the costs against the amount and quality of data obtained.

So why might online user testing not be right for you?

Reason 1 – Less time = fewer tasks = less data = fewer insights

The average user session at lasts only about 15 minutes, severely limiting the amount and length of the tasks in the scenario. Compare this to a DIY solution, where a user is more than comfortable spending an entire hour undertaking 2–3 lengthy tasks in addition to a homepage ‘5 second test’.

Reason 2 – Is it really the cheapest solution?

A couple hours of your time and incentives for users are the only costs of doing user testing in person. Additionally, if your target audience is pretty general, you can always find some colleagues or friends who would do it for free or a pint of beer.

At three users will cost you $117 (or $195 for five), plus bear in mind you’re likely to require twice the amount of participants compared to DIY (see Reason 1).

Reason 3 – Lack of recruiting control

With only demographic targeting available, online user testing solutions are basically limited to generic sites only. Any niche website or serious B2B requiring some domain knowledge from the user would never be able to find suitable representative.

There is always an opportunity to specify additional requirements but when the decision as to whether a user suits the requirements is made by the user himself I would really question the level of accuracy.

Reason 4 – Professional test participants

Generally the likelihood of your participant being recruited for a particular test (e.g. yours) depends upon their ‘rating’. This rating is given by the test creator. Logically the system chooses users with higher ratings much more often than the rest, unintentionally creating small set of ‘professional users’ participating in plenty of tests on a regular basis. This creates at least two major problems:

1. Experience gained by these users from large number of user test sessions dramatically changes their behaviour towards ‘expert review’ style rather than ‘user thinking out loud’. They seem to focus on reviewing lots of tiny general practice roadblocks when we just need a user following the scenario providing us behaviour insights, not an opinion. There is nothing worse than an amateur giving expert advice.

2. Although maybe not experiencing the fear of hurting a test creator’s feelings that occurs in DIY tests (especially if there is an incentive), online user testing participants users are interested in a higher rating from the test creator and often will do almost everything to supply you with a set of minor, easy-to-tweak insights rather than honest severe leaks that usually could question your marketer/designer/developer professionalism.

Reason 5 – Lack of session control

This is without doubt the biggest reason to hate remote user testing, and it still pushes me to advice offline one-to-one user testing for most clients. From my experience at least 25% of every online remote test I conducted was ruined because of this issue. You can spend days on writing clear scenario and intuitive tasks but there’s always going to be some people missing the point of the task, simplifying it or thinking they completed it despite being only half-way through.

It becomes more of a problem when we consider ‘quiet thinkers’ – users continually forgetting to think out loud and only describing what they do. In an offline solution there’s always the opportunity to ask “Why are you doing this?”, “What are you thinking about?” and “What are you planning to do next?”.

And to finish…

Even if the reasons above might sound critical, I still would not recommend excluding online user testing from your tool box. What I wanted to do with this post is to set up some basic factors influencing your decision on which method to take.

I’ve no argument that, if you are limited with time or want to test small part of your conversion process, should be you BFF. On the other hand if quality of insights is your main priority consider running test by yourself. It might sound complicated but the pay off will be worth it.

Finally, if this all sound way too complicated simply get in touch with us and we will take care of entire user test process for you including analysis and hypothesis preparation.

Happy testing!

Leave a Reply

Your email address will not be published. Required fields are marked *

4 thoughts on “Online User Testing | 5 reasons to avoid it”

  1. Chris says:

    Thanks for your honest comments Kiril. There are certainly scenarios where unmoderated, usability testing does not work. If you’d like, please contact us in the future and we’ll help you write your tests using best practices we’ve learned over the last several years (to avoid many of the blocks you’ve mentioned above)

    • Tim Barlow says:

      Hi Chris,nnThanks for coming by. It’s great to get an “official” response so quickly. I’m sure Kiril will be keen to work with you to try out your techniques the next time we use I’ll look forward to the follow up post in due course.nn(Chris – you asked if we were able to retrospectively get your url included in your profile on the above comment to prove who you are. Sadly we can’t but we are honoured to have you, the VP of marketing commenting here. And here’s a link to your LinkedIn profile )

  2. Claire says:

    We’ve only just started online user testing. Have to say we’re very impressed with the product and service. I’ve witnessed a $20,000 user test in progress and to be honest I’d say the $400 online test was streets ahead. For what it’s worth, I think reason 5 in this article is the very basis of why you should do online user testing…..losing control is part of the fun and gives and honest look at your site and raised issues we hadn’t thought of. Also re: Reason 3, we used our own testers and they signed on through the online testing service. This gave us 100% control in making sure we were able to test our brand with the locals.

  3. Kiril says:

    Hi Claire,

    Thank you for your comment. It’s been more than a year since I wrote this post and things got severely improved during this time. We are still using almost on a daily basis and I can definitely see massive improvements in quality of insights we are getting now.

    Regarding the reason N.5: I don’t have anything against loosing a bit of the control on the users’ journey, the lack of control I mentioned was regarding the quite thinkers or complainers. Users who are either silent 70% of the time or those who constantly complain if they can’t complete a task within few seconds. I’m actually seeing more and more of the complainer types where users can’t be bothered thinking and behave like they are giving you a favour.

    Recruiting your own users is probably the most optimal solution which would enable you to avoid most of these reasons.

Like the Brain? Sign up for the packed-full-of-tips monthly newsletter