The survey data we obtain from an effectively designed questionnaire can be extremely useful in answering some research questions, but identifying when surveys are not the most helpful implement in our toolkit is also vital. Having explored some of the main reasons why surveys can be useful in user research in Part 1 of this series, let’s think about when you should use them.
1. Descriptive value
Surveys are extremely useful for describing the characteristics of large populations – the ‘who’, ‘what’, ‘where’, and ‘when.’ They are less adept at allowing us to explain people’s behaviours – the questions of ‘why’ and ‘how.’
Some surveys combine closed questions (which have a pre-set list of answers for respondents to choose from) and open-ended questions (which allow participants to write a free-form answer). While closed questions don’t afford the respondent with the opportunity to provide context or explain their answer, open-ended questions do allow for more detailed, nuanced responses.
However, it’s important to recognise that surveys have limitations and shortcomings. Without the ability to ask follow-up questions, prompt for further information or ask respondents to clarify what they mean, surveys are limited in their capacity to explain an individual’s attitudes or beliefs.
To get a full picture of what’s going on in a complex problem, it’s often best to employ a mixed method approach, combining quantitative and qualitative data collection and analysis, so that you can gain a detailed insight into all of the questions above – the ‘who’, ‘what’, ‘where’, ‘when’ and the ‘why’ and the ‘how.’
In that sense, they can be used very effectively before conducting qualitative research, because you can use the descriptive picture obtained from the survey data to inform the next phase of research.
2. Hypothesis testing
Often when we have a problem to solve, or we have a set of research objectives, we might have a theory about what we’re expecting to see. When it comes to testing those hypotheses, we want to make sure they are empirically grounded, and tested using real world, observed evidence.
Whether a survey is the best way to obtain that empirical evidence depends largely on your hypothesis. Surveys can be a great way of testing whether your hypothesis applies broadly, and not merely in a few specific occurrences.
For example, you might have conducted a series of in-depth interviews or focus groups with participants, in which the same patterns keep emerging. Several individuals may have voiced the same frustrations with a particular service. Unless you speak to a sufficiently large proportion of people in that population however, you can’t be confident that their experiences are typical of the majority. In these scenarios, a survey could be used very effectively after a phase of qualitative research.
3. Discovery
Surveys can be a useful tool for quickly finding out some of the more descriptive information you want to understand in the discovery phase, like who your users are. This could include any demographic variables you need to know (like their age or geographical location), or the range of abilities your likely users have.
A well-designed questionnaire can also help you gain some understanding of the most common barriers, pain points and frustrations users face, as well as the priorities and needs of different types of users.
To gain a full understanding of these however, and produce a detailed map of your users’ experiences, a survey alone is unlikely to provide you with enough context or explanation. When trying to understand what your users do now, how they do it, why and how they face certain problems, and more complex aspects of who they are (like their attitudes and beliefs), it is important to use qualitative research.
4. Alpha
Alpha is probably the phase in which surveys are least useful to user researchers, as the deep, end-to-end understanding of users’ lives and work we need during this phase is usually better served by qualitative research methods. Similarly, surveys are limited in their use when it comes to detailed testing of design concepts and interactive prototypes.
Sometimes during alpha however, it is helpful to quickly gather feedback on design ideas from a large number of users. Surveys allow you to reach thousands of people in a short space of time and gauge how they felt about different design ideas. If this is done at the beginning of the alpha phase, you can also collect their contact details if they would like to take part in further research, making it easier to recruit participants for the qualitative sessions you conduct later in the phase.
5. Beta
Some of the information we need to find out in beta is more suited to qualitative research methods. For example, face-to-face and remote usability tests are usually the most appropriate ways to get a deep understanding of usability and accessibility issues.
In beta, surveys can be used to quickly collect detailed feedback on the developing service from many of its users. If you begin your user research in the beta phase by surveying your users, this can help you identify the most common usability and accessibility issues, which you can use to inform the qualitative research you do later in the phase. Like in alpha, you can also use the survey to help find and recruit participants for face-to-face or remote tests.
In Beta and Live, services often add a static link within the digital service as a way for users to provide feedback at any time. This can help you identify areas for continuous improvement for a live service. The drawback is that the sample is self-selecting and would only be from users who are already able to engage with a digital service. For your non-digital users, you need to make sure you use another method too.
6. Live
In the live phase, the aims of user research are to assess people’s experience of using your service, understand evolving user needs, and test new features, changes or improvements to your service.
To some extent, conducting surveys can help with all of these aims, but they are most effective when used alongside other research methods. For instance, analysing support tickets can help you identify problems users have with your service, but these often lack the detail and context you need to fully understand their pain points and frustrations. You can use surveys to collect much broader feedback and to reach a much larger sample of users.
Once you have identified the issues users tell you about, you can use interviews, visits and usability testing to get a deeper understanding of the users’ problems and how their needs may be changing.
Limitations in the survey design
Many of the disadvantages and limitations we face when using survey data are related to the design of the survey itself. In some cases, there may be unavoidable limitations, which is okay if you recognise them, identify them, and acknowledge them. Often, you can account for these limitations by including other forms of data collection in the research method. In other cases, limitations can be mitigated by following best practice guidelines.
Up next: best practice in survey design
Having looked at why and when surveys can be used most effectively in user research, the next three posts will focus on how to design surveys, using best practice. The next post will focus on how to effectively plan and structure a survey, to provide the best possible experience for the research participants, and ultimately gather the most useful, reliable data we can.
To find out more about user research at Mastek, reach out to me at michael.watson@mastek.com or on LinkedIn.
