How to rate the ratings

Surveys where participants are asked to rate different factors can be a useful tool in gathering data. But the way they are presented could leave them vulnerable to bias because of the different ways people perceive them, a new study has found.

Participants took part in an aircraft simulator task then rated their mental workload and situation awareness. Mental workload was rated using a NASA-TLX questionnaire – which uses polarity and 21 scale divisions – while a numerical SART survey which has seven divisions, was used for situation awareness. The questionnaires had been adapted so the rating scale featured either polarity, numbers or a colour gradient.

Out of the 82 participants, 29 preferred the numerical scale, with 24 picking the colour gradient, 21 choosing the polarity option and eight saying they didn’t find there was a difference between the different versions.

Some participants found the use of the colour red in the gradient scale ambiguous as it wasn’t clear which extreme it was meant to represent. The same issue was raised with the numerical scale where people weren’t clear if 1 referred to success or failure.

The study, presented by Niall Miranda at Ergonomics & Human Factors 2024, said: “The use of numbers and colours in a rating scale is superficial due to subjective interpretation of the numbers and colours. An ideal rating scale used in a pen-and-paper questionnaire would consist of numerical divisions in combination with an assignment of polarity on the ends of the rating scale.”