CX Survey Design Series: Which Direction Should A Scale Go?

 

By George Polyard

Download
mobile device with survey

Introduction

 

Surveys are the backbone of customer experience (CX) programs. Businesses today, from international hotel brands to small independent retailers, have systems in place to measure customer sentiment with surveys often being the foundation.

In a survey, it’s recommended to start with the worst (or lowest) option on the left. The opposite can skew your data positively.

When designing a survey there are many perspectives on what scale to use. Different scales, such as 5, 7, 10, and 11-point, all have their pros and cons, not to mention the blunt thumbs up/down scale of many review sites (we’ll leave that for another day). After selecting the scale type, the next question is how it should be oriented. That is, should it have the lowest/worst score on the left ascending to the highest/best score on the right, or should it begin high/best and descend?

survey scale examples

Survey design, including graphics, layout, and verbiage, plays a significant role in both the resulting data and the response rates. CX practitioners typically prioritize these metrics, and the directionality of scales should not be overlooked.

Psychological Factors

 

According to a deep dive on this topic from Boise State University, there are several factors that contribute to how a survey scale may be interpreted. First, there’s the left-side selection bias, which is a tendency toward options at the beginning of a list called the primacy effect coupled with the fact that most languages are written left to right. Second, there’s some social In a survey it’s recommended to start with the worst (or lowest) option on the left. The opposite can skew your data positively. psychology at play here in the form of the satisficing theory, where people are biased toward options that minimize their personal psychological costs, and the social desirability bias, where people are biased toward options that are more socially acceptable or desirable.

When combined, these theories bias options on the left, options that minimize personalpsychological costs, and options that are more socially acceptable. This suggests that survey responders are more likely to select the leftmost options, especially if they are the positive options.

Research

 

Since 1966 there have been at least 13 scientific studies focused on this topic. The studies have occurred in various countries and used several different languages. Various age groups have been included and both written and verbal surveys were analyzed. The aggregate result: if a survey scale is descending (starting with the highest) it will typically have a higher score for the same question than an ascending (starting with the lowest) scale. In most of these studies, the researchers found higher ratings when using descending scales and recommended the use of ascending scales. For example, a 2015 study by Hotaka Maeda at the University of WisconsinMilwaukee administered a 15-question online survey to 1,693 adults. The resulting average score of surveys with a horizontal descending scale was 3.03% higher than ascending.

A NOTE ON RELATIVE VS. ABSOLUTE JUDGEMENT

A 2006 study from Nicholls, Orr, Okubo, and Loftus, called out an interesting consideration. If your survey is used to exclusively make relative judgments (comparing one internal department to another), then the scale direction doesn’t matter, because if there is a bias towards one side of the scale, the bias would apply everywhere. However, if your survey is used to compare your score to others, such as your competitors, then the scale is an important consideration.

Consistency

 

To not confuse the survey taker, the direction of all scales should remain consistent throughout a survey. Though the scale itself may change (5 points vs. 11 points) the direction should not. Additionally, once a survey program is active, the direction of the scale should not be changed. As previously stated, the order may impact the reported data. If you desire to compare your scores year-over-year, be intentional and careful when changing the scales of an ongoing program.

Practicality

 

Perhaps more significant than the aforementioned psychology is the decision that most CX programs have taken. A completely unscientific look through many surveys, including heavy hitters such as Apple, Saks Fifth Avenue, and Marriott, confirms that an overwhelming majority use ascending scales. (Try for yourself by searching your inbox for “survey” or “feedback”.) The widespread use of ascending scales primes survey takers and makes them accustomed to one order. As such, the use of descending scales will likely confuse them. That is probably why, to most people, Figure 1, below, looks correct and Figure 2 looks odd.

survey scale examples

Further, many of the studies referenced in the 2018 Boise State University research recommend the use of vertical scales, as they were found to reduce the impact of scale order. However, in practice, the use of vertical scales greatly reduces the number of questions that can be displayed simultaneously. Additionally, when used in an online survey, vertical scales require excessive scrolling and increase the time it takes to complete a survey.

A final solution was mentioned in several research papers on this topic, to use the mean of two types of surveys: one with ascending scales and one with descending. Though this may work in some settings, such as less frequent ad hoc research, the practical application of this would likely be too complex for most practitioners and survey software vendors.

Conclusion

 

The decisions around how to design a survey can be a very personal decision for a brand. Surveys, though ubiquitous, are an effective way to communicate with your customers, demonstrate to them that your brand cares, and obtain objective actionable data. CX practitioners are often confronted with a peanut gallery of opinions on how a survey should be constructed. Research-based evidence, such as guidance on the orientation of a survey scale, can not only resolve survey design debates efficiently but result in more effective surveys with more accurate data.

Reference

 

Chyung, S. Y., Kennedy, M., & Campbell, I. (2018). Evidence-based survey design: The use of ascending or descending order of Likert-type response options. Performance Improvement Journal, 57(9), 9 – 16.

LET'S GET TO WORK