My father is a retired cabinet maker – during the COVID lockdown, it was more evident than ever that I didn’t inherit his furniture making skills. But as a child when I did potter in his workshop, and asked to use one of his many tools, his answer was often a challenge for me.
For example, I would ask if I could use a chisel, and he would answer with – “which one?”
His knowledge and collection of chisels was way beyond my abilities, so I would describe what I wanted to do, he would interpret that, and may say, “oh, you need a paring chisel.”
He selected this chisel from many options, which could have been a dovetail or carving chisel, or any one out of his extensive collection. Some days he would say, “Are you sure you want a chisel, you probably need a plane.”
Choosing the right question to achieve each survey task is equally important when designing surveys.
The proliferation of cheap online survey software over the last 15 years has meant that now everyone has access to a survey tool box. But a tool box doesn’t automatically lead to success – the most valuable tools in research are your questions, and knowing which question types to use and when, is what can make all the difference.
Here’s a simple example of inadequate question selection that I have seen recently.
An organisation wants to find out which features are most important to its customers/citizens from a list of possibilities. They compile a long list of features and ask respondents to assess each on a scale from ‘very important’ to ‘very unimportant’. The survey may state the real-world realities to respondents, for example a low price-point and high quality, to encourage trade-offs to be made. But if these trade-offs are encouraged, but not enforced, they are meaningless.
From this survey, a result will be achieved but its relevance to the real world will be weak. This is because the question used is not a sharp enough tool.
Specific problems with a long-list question approach are:
Long lists are mentally challenging and time-consuming, so respondents give up
Some concepts are harder to understand than others, so respondents don’t select them
Items at the bottom of the long list are less likely to be selected
When respondents are asked to make ‘trade-offs’, but aren’t forced to, a large, vague list of priorities is identified
In some instances, the long list still won’t be exhaustive, so items not included aren’t considered – in this case, a list isn’t the best tool.
A key test is, can you repeat the survey with a similar group of respondents, and get a similar result?
In many instances the result will be quite different, proving that the relationship between the result and the opinions of survey respondents is weak. The result will be at best inconclusive and at worst, misleading – and getting it wrong can be costly for a business or a city.
The solution is knowing your tools and selecting the right one for each of the questions you need to answer. Similar to making a piece of furniture, a survey will normally require more than one tool to achieve the desired result. You need to know what tool to select, and how to use it at each stage of your survey.
For the example above, there are three alternative options:
Use a conjoint survey approach, which forces respondents to make trade-offs between sets of features and through this, identify how respondents prioritise the various features. There are many conjoint types to choose from.
Break your long list into smaller sub-categories and then test the importance of each category, by asking ranking questions.
For very long lists, don’t ask for importance. Instead, use them to stimulate/prompt thinking and follow up by asking respondents to state what they think in their own words. Open-ended questions can be scary, but they can also provide a level of insight that closed questions often can’t.
No matter the size of the task, from identifying what are the most sought after product features for consumers, to rebuilding an entire city, key to capturing quality information to inform decisions is using the most precise tool.
The next step is asking sharp questions; like making furniture, your tools need to be sharp to be accurate. But like the outcomes of my furniture making – that is for another day.
Cover image source: Photo by Todd Quackenbush on Unsplash
Inset image source: Photo by Barn Images on Unsplash