Planning and structuring a survey

While surveys aren’t difficult to create or distribute, a badly designed survey can bias, confuse and mislead your respondents. In the third part of this series, I will focus on how you can effectively plan and structure a survey, to provide the best possible experience for the research participants, and ultimately gather the most useful, reliable data you can.   

In part 1, I looked at why surveys can be useful in user research, while part 2 considered when surveys are particularly useful. 

This advice is based on a combination of guidance from the Government Digital Service (GDS), the Market Research Society (MRS) and the Royal Statistical Society (RSS), as well as a range of academic sources and personal experience.

Your survey’s structure should: 

1) …be easy to use  

Your survey should be easy for your participants to follow. Ideally, you should plan a questionnaire to make sure the structure is logical from the point of view of the participant.  

Before you ask people any questions, you should introduce your participants to the themes and topics which your survey will cover. You should also make sure they know what your research is and what your aims are so that they can make an informed decision about whether they want to take part.  

Some surveys contain a ‘routing plan’, where questions shown later in the survey depend on how they answer earlier questions. It’s a good idea to outline your structure and routing plan before you even write your questions.  

It’s also important to note that the structure of your questionnaire does not need to follow the order of your objectives. If a project has several key objectives, it’s tempting to use those to structure the survey, so that you (or your client) can see you’re answering all of them.   

It's far more important that you focus on making the structure flow in a logical, usable way, even if that means spending more time explaining where you’ve covered all those objectives in the survey.  

2) ...flow naturally   

Whenever possible, your survey should follow a natural flow, so that it resembles a logical conversation or train of thought.  

Next, you might start with a few demographic questions, to find out a bit about who your users are, before you move on to more specific questions about their experiences.   

Much like when you first meet somebody, you would generally get to know them before you asked them something particularly personal or sensitive, it is a good idea to avoid complex questions at the beginning of your survey. If you start with some non-controversial questions which are easy to answer, this can ‘warm up’ the respondent, so that they are more used to answering questions by the time you ask them something more complex. 

The order in which questions are asked can have an impact on the way they are answered, so sometimes you might want to consider randomising the order in which some of your questions are asked, so that different groups of respondents are shown them in a different order. Randomising the question order is a perfectly legitimate way of reducing this risk, but only if the structure still flows naturally.  

In order to keep the flow natural, you should try to avoid non-sequiturs - statements which appear to have no relevance to what came before them. In some cases, it might not be possible for you to avoid any non-sequiturs, for example in an omnibus-style questionnaire, where a large number of respondents are asked about a variety of unrelated topics. In these situations, you should use appropriate introductory phrases, so that they don’t feel confusing or random.  

3)…include different question types  

To prevent what’s known as ‘respondent fatigue’, it’s a good idea to use a variety of different question-types to break the routine within the questionnaire and avoid repetition. Respondent fatigue is the phenomenon which occurs when survey participants become tired of the survey tasks and the quality of the data they provide begins to deteriorate.   

You can design a brilliant set of questions, which will theoretically answer all your research questions. In practice though, the respondent might become bored and their attention might start to wander. If this is the case, the reliability of their responses will be affected and the rigour, integrity and quality of the data you receive will be severely hampered.   

The rigour, integrity and quality of the data should be at the forefront of your mind throughout the survey design process, so wherever possible, you should look for ways to make your survey engaging and maintain the participant’s interest.  

4) …avoid being too long   

For the same reasons outlined above, you should avoid making your survey unnecessarily excessive in length or ‘over-surveying’ your sample. If certain questions probably aren’t going to be analysed, you need to think about whether they’re adding enough to value to warrant their inclusion.  

If some questions only really apply to certain segments of your sample (e.g., a certain demographic, or people who have answered previous questions in a certain way), then you should use a routing plan so that you only ask those questions to those participants. This will make more work for you in the short term, but it can help you avoid confusing the respondent or wasting their time and will reduce the risk of respondent fatigue.  

Progress bars, which allow participants to see how much of the survey they have already completed, can also be very effective in mitigating respondent fatigue. Whether you’re able to include these or not will depend on the survey platform or provider you choose to use.  

Up next: how to effectively write a survey  

Having looked at why and when surveys can be used most effectively in user research, as well as how to effectively plan and structure a survey, the fourth part of this series will consider best practices when writing the survey itself. 
 
To find out more about user research at Mastek, reach out to me at michael.watson@mastek.com or on LinkedIn.