User Testing: Hard Won Experience

Reading tips and guides from others can really help with getting to grips with a new process. The most valuable insights, however, come from doing specific tasks hundreds of times, making thousands of mistakes and learning from them. That’s what people usually call “experience”!

This is certainly how it has played out in my journey of getting to grips with User Testing. I have read books from Steve Krug, dozens of white papers and hundreds of blog posts, all of which pushed me to start user testing by myself and last week I finally conducted my 100th user test session. Some of them were very successful, others not so and few were a complete waste of time. That’s why I wanted to share with you all the mistakes I’ve made, so you can avoid making the same mistakes if you are just starting out with User Testing.

User testing is a technique used for evaluating websites by testing actual users. Being more precise, it’s sitting a real person in front of your website and videoing them trying to complete several tasks on your site (usually with much greater difficultly than you would have thought!)

Switch on the recording

You would not believe how many user testing sessions went wrong just because the screen recording was not on! Just remember that a user test without a video recording is almost useless, because you never would be able to restore any insights from your memory or even dozens of notes a couple of days later. Better interrupt the user to double check if you are not sure, rather than finding it out at the end of the session.

Don’t talk

User testing is definitely a qualitative type of research, meaning that entire session should be focused on getting as much qualitative data as possible. That’s why any comments or approval gestures from you will influence the behaviour of the participant and essentially ruin the test. Remember, the aim of the test is to see how your real visitors engage with your website. Let them engage alone, without you behind them approving or disapproving each of the mouse moves.

Don’t answer questions

User participants sometimes take the responsibility of their “test” very seriously. Lots of them would be continually asking questions about the task: Where should they start? Am I doing it right? And so on. Similar to the previous rule, you never should answer any of these questions. If a user insists, just say you can answer everything at the end of the session. Your primary focus is the actual user behaviour, the same user who would browse your site from home sitting on the sofa. And these users don’t have personal user experience advisers at home. At least most of them don’t.

Don’t take notes

This rule might sound very controversial. I used to take brief notes of some interesting moments before, but then I found it more ineffective than useful. Here are two reasons why:

1. It’s a waste of time. The video recording is capturing all the required data for your further analysis. I never had a single case where my notes had any useful insight which I couldn’t get from watching the video from a session.

2. It affects the tester’s behaviour. It’s something similar to the job interview or the oral exam. When you see a person who was fully listening to you beforehand, suddenly starting to write down some long notes, you obviously think you just screwed up somewhere. The same with user test participants. I’ve seen enough times a user considerably changing his already started journey, just after I wrote down something on a paper. Well, it’s almost impossible to prove I am right on this point, but if you know there is even 5% of testers whose behaviour would be influenced, would you take that risk?

Don’t abandon your tester!

Even though first two rules were about minimising contact with a user during the session, sometimes you just need to interrupt. If you see that the participant is completely lost, if he is trying to complete a task on the entirely wrong page or seriously confused – don’t waste the time. If it’s just a beginning of the task, point them towards the right solution or a page and let them continue from there.

If it’s the last stage –just move on to the next task or the next participant. Don’t get too upset. First of all, have a look to the task description again and make sure it’s clear and understandable. Secondly, if something like this happens you already can conclude a session as successful and think how to solve the problem later. The main point of user tests is not to prove that everything is perfect but to find these exact cases there your site sucks. Find the solution later and enjoy the boost in conversions.

Keep track of time

One of the hugest advantages of user testing is that you can run at least 3 tests in one morning. In order to do it you always should estimate the time required for your participant to implement a task. Remember, a user session consists of an intro speech, a landing page first impression test, some background info and 3 tasks. And you have around an hour for all of it. Make sure you maximise the time on the tasks and distribute the task time evenly over each one. However always start with the most valuable one so if you do overrun, you can just skip the last task.


This rule requires a lot of practice to master. If you are on your first dozen user tests don’t worry about this rule.

From time to time you will end up with a badly chosen test participant. He or she will simply not be able to give you any valuable insight, usually because of a lack of so called “domain knowledge”. This will usually become apparent during the first task and pushing on with the second and third will provide you with no benefit and will make the participant increasingly uncomfortable.

For example a case from my own experience: a user failed to find the relevant solution for his business while we were testing a web app developer’s website just because his knowledge of internet solutions, goals and methods was close to zero and limited to e-mail only. There was no sense to go through the rest of the tasks (focused on choosing the right price plan, trying the demo and finally making a purchase), so I replaced them with more generic ones, focusing on content findability rather than decision making, requiring no domain knowledge but still providing some qualitative info about the user experience.

Rather than waste the time, transfer your session to less knowledge requiring tasks: search for a similar company case study, question from a FAQ section, search for a review and so on.

Keep them focused

People love to talk. Especially when you show them so much attention by inviting them, being friendly and asking for help. You can’t even imagine how many times my test participants were deciding that right in the middle of a 5 steps task, it’s helpful to share a joke. Or to ask for advice for their cousin’s plumber website.

Make sure to explain the importance and time limitation of the session at the beginning. Mention at least a couple of times the negative value of interruption and promise to answer questions, honestly laugh at their jokes or do the SEO audit for the plumber cousin at the end of the session.

Test a user, not an expert

1. You get usability/website advice from people far less qualified than you. 99 times out of 100 the advice will either be plain wrong or something that was obvious to you anyway. Keep your participants focused on the problem, not the solution.

2. To get valuable insights from opinions you need the views of many more people than you will be working with in user tests. A colour that annoys one will sooth another. A price point that seems reasonable to one test participant, may feel ludicrous to your next participant. Surveys and split/multivariate testing are much better ways to tackle these sort of issues. Keep your user tests focused on identifying issues that are making it difficult to complete objectives.

As a facilitator you should be ready to filter opinion out. Keep asking “what are they thinking about?” not “what’s your opinion on X?” Moreover, you should clearly explain this to your users at the start of the test.

These are the biggest mistakes I made. If you have had a crack at user testing I am sure most of you will have lots of others. Please do not hesitate to share in comments and I will promise to add them to this list with proper referencing to the author.

Alternatively if you would like some help with running low cost user tests on your website, I’d love to hear from you.

Happy Testing!

Leave a Reply

Your email address will not be published. Required fields are marked *

One thought on “User Testing: Hard Won Experience”

  1. Tim Barlow says:

    Welcome to the team, Kiril. Looking forward to some fantastic insights on conversion rate optimisation in the Attacat Brain

Like the Brain? Sign up for the packed-full-of-tips monthly newsletter