Online Surveys: 4 tips on how to waste a budget

Newseum: Do You Trust Blogs?I had an idea to write some common guidances for the online surveying process a long time ago but whenever I was putting down some notes I found myself with a list of very intuitive common-sense advice, and playing the role of Captain obvious is never the goal of my posts.

However last night I stumbled upon the survey offered on behalf of a massive international enterprise on their website and funny enough, within the first 10 minutes of filling in the survey, I spotted 4 issues I had thought were easy to avoid for everybody. If one of the most recognised brands in the world, with an enormous budget and army size team of marketers, have succeeded at combining almost all possible mistakes in one survey then perhaps the pitfalls are not so obvious any more.

4 tips on how to waste a budget with an online survey

1. Lie to your visitors

9 out of 10 surveys state the amount of time it takes for a user to complete; fair point as you always should state how much effort you require from your visitor so they can compare the benefits of completing the survey with amount of friction requiring.

The survey I mentioned clearly stated it would take 5–15 minutes of your time (unfortunately I forgot to take screenshot and now can’t access it) when in fact it took me at least 25 and, if I were to read every question thoroughly and spend some time on thinking, this number could easily double.

Is it so hard to ask your friend/colleague/cleaning lady to complete the survey in front of you so you can track the actual time required?  What do you think a person who was promised to waste 5 mins of his time would do after finding himself spending 30 min and still being far away from the end?

The same rule applies to showing the ‘% complete’ figure; why does answering 3 simple demographic questions show that I’ve completed 23% so far, but then subsequent multiple-choice and really time-consuming questions add only 3%? Why not just state the number of questions left? It’s clear and easy.

(click on the image to enlarge)

These issues don’t just result in very high-abandonment rates or plenty of unqualified participants (quite often your users would just randomly click on any answer in order to complete as fast as possible), they also seriously damage the brand of the organisation this survey was created for. Why should I believe that an organisation’s service or product delivers when they just shamelessly lied to me in order to get the info they need?

2. Ask questions you already have answers for

At least half of the questions asked in the survey could be answered just by spending 2 minutes in whatever web analytics solution you use.

Some version of this can be spotted in almost every online survey – usually the reason behind this mistake is broken communication between a company’s departments, or commonly they’ve employed traditional old-school market researchers to do the survey online. I am not saying traditional marketing research is ineffective; all I want to say is that old-school marketers rarely have any idea on what is currently possible within the internet marketing industry.

(click on the image to enlarge)

And you don’t need expensive corporate web analytics solutions (usually used by huge companies as in our example, often starting from 10k per annum) as your simple Google Analytics would answer the questions without any segmentation or customisation.

(click on the image to enlarge)

In this case integrating a research team with the analytics team would not just answer half the questions, simplifying the process by a mile, but would produce credible and reliable insights in place of doubtful users’ opinions.

3. Ask useless questions

This point combines both way-too-general questions like:

(click on the image to enlarge)

Rate exactly what? User experience? Colour choice? Web copy style?

Always make sure to specify exact information you are looking for. More specific questions will eliminate “Don’t know” variance and mostly importantly will actually highlight the problems rather than give a really opinionated mark.

And questions not providing any insights:

(click on the image to enlarge)

First of all, please have a look at the questions “This website inspired me to do something new” and “The website reflects British Culture”. What kind of British culture or inspiration are we talking about? The site is a web-player-style tool where you can catchup with TV and radio shows you missed. It’s just a tool not a guru.

What about the slightly differently-worded questions like “The website is up to date” and “The website is fresh and new”? For sure it is. It’s a website providing opportunity to catch up with recent shows you missed. Being up to date is the primary function, otherwise it would be completely useless.

Every single question should be built up as a micro-hypothesis. In simple terms you should have a clear idea of the precise actions you are going to take in case any answer shows statistically significant popularity. If you are not able to do that then forget about the question. The primary purpose of your questionnaire is improving user experience based on the findings, not a 70 page dust-collecting report on a how a web player represents British culture.

4. Make the process look complicated

What would you think if you saw something like this?

(click on the image to enlarge)

Any user would quietly swear after seeing something similar. Is it so hard to divide it into 3 parts? Visually heavy elements will always look more complicated than they actually are, so why multiply already massive friction? Adding 1 or 2 more steps would not make any severe difference to the already long and time-consuming process but could ease the user experience by a mile.

Make sure to make every step of your multi-stage questionnaire easy to digest and scan. It’s not  the number of steps responsible for friction, it’s the amount of time and effort it takes to complete.

Follow up

If you are willing to learn more about the benefits of using online surveys (I can be blunt) as the part of Conversion Rate Optimisation join me this Friday at New Media Breakfast or subscribe for our webinar.

Leave a Reply

Your email address will not be published. Required fields are marked *

4 thoughts on “Online Surveys: 4 tips on how to waste a budget”

  1. Perhaps the generic user data is to give the respondent a profile? IE I’m sure they know how many people use the site, but knowing how much a respondent uses the site will probably affect their answers to other questions.nnAlso, you appear to have taken steps to hide the brand in question, yet all the images are called BBC-Opinion-Survey etc and the last graphic has a number of iPlayer references…? Is that intentional?nnEDIT: PS I should also add that it is a very useful post – it’s amazing that so many online surveys are so terrible! These guys need to get @attacats on the job! You should follow this up with them!

    • Kiril Bunin says:

      Regarding the profile:nI thought so too in the beginning but after couple of various submissions understood that it’s not. Only age question was filtering out all way too young users. Anyway, even that would not be an excuse because it’s easily possible to assign profiles based on Analytics data as well. And with the budgets this type of companies usually operate it is a must.nnRegarding the brand:nnNot intentionally at all, I noticed that only after your comment. This blogpost was written and published almost simultaneously with all our “office move activities”, so my attention was seriously damaged.nn I didn’t want to slag off anybody’s work or brand, even though I think it is done on a quite amateurish level. The only reason I mentioned big corporation is to strengthen the importance of really obvious mistakes. If I would write about local Bob’s bakery than this mistakes are understandable based on lack of knowledge or budget. But for the company of this size and investments these mistakes are unacceptable. Epic Fail.

      • Interesting – when I edited my comment, it posted a new comment rather than updating my original one!nnProfile – sure, I appreciate we’ve not seen all the questions so thought it was worth mentioning.nnBrand – quickly edit the photos and delete my comment… no one will ever know!

  2. Perhaps the generic user data is to give the respondent a profile? IE I’m sure they know how many people use the site, but knowing how much a respondent uses the site will probably affect their answers to other questions.nnAlso, you appear to have taken steps to hide the brand in question, yet all the images are called BBC-Opinion-Survey etc and the last graphic has a number of iPlayer references…? Is that intentional?

Like the Brain? Sign up for the packed-full-of-tips monthly newsletter