Response Options

There are several decisions to make that involve response options. How many response options should you use? Should you use an even or odd number of response options? Should you label them? This section contains a summary of best practices that one can use to address these questions.

Number of Response Options

The question of how many response options to use centers around two main concerns. The first is that more options means you can obtain a more fine-grained assessment of the characteristic that is being evaluated (e.g., attitude). In other words, your assessment is more precise. However, the question is how the number of options affects the reliability and the validity of the measurement. With more options, it becomes more difficult for people to distinguish between the different options (e.g., is “Strongly agree” reliably different from “Very strongly agree”?).

Table 1 shows an overview of various studies in which the topic of response options was addressed. The studies vary in many ways, so the final conclusion should be a holistic interpretation of the results, rather than a simple tallying of the results. Note also that only empirical studies are included and not simulation studies. Simulation studies seem limited because they cannot address the plausible psychological limitation of people being unable to distinguish between many options.

There are also several review papers on the topic. Krosnick & Presser (2010) suggest that 7-point Likert scales are probably optimal. Lietz (2010) concludes a desirable Likert-scale consists of 5 to 8 response options. Similarly, Cox III (1980) recommends to use between 5 and 9 response options. Symonds (1924), in 1924, claims the optimum number is 7. Gehlbach & Brinkworth (2011) recommends using 5-points for unipolar items and 7-point for biopolar items.

There are also statistical arguments for why a particular number of response options is preferred. With more response options, the assumption of normality is more likely to be tenable. Some of the papers included in Table 1 (e.g., Rhemtulla et al. (2012)) are about this concern.

Besides psychometric properties it may also be worth taking into account respondent preference. This involves ease of use of the scale and whether the response options allow for sufficient variation for respondents to express their view. Preston & Colman (2000) found that respondents found scales with 5, 7, and 10 points easy to use (compared to fewer options and a VAS) and that they preferred scales with more response options to allow them to express themselves (7 or more). Other studies also show that respondents favor more options (Cox et al., 2017).

Note that if time is of the essence, fewer response options are preferred.

Another relevant factor is whether the scale is bipolar or unipolar. Bipolar scales are symmetrical which means the number of options naturally increase as they need to match both sides of the spectrum. Unipolar items are only about one side, usually ranging from the absence of something to the presence of something (to a certain degree). Since it is harder to label a larger number of options for a unipolar scale, the number of options are likely to be smaller.

Conclusion: It appears that few response options (2 or 3) should definitely be avoided. More response options therefore seems better, but benefits seem to quickly level off. Given other concerns, such as ease of use and interpretability, a 7-point Likert scale seems to be preferred for bipolar scales and a 5-point Likert scale for unipolar scales.

Odd vs. Even Response Options

The middle option of a scale can have an ambiguous meaning. Participants may use it to indicate a moderate standing on the issue (Rugg and Cantril, 1944), a lack of an opinion (Nadler, Weston, and Voyles, 2014), ambivalence (Klopfer and Madden, 1980; Schaeffer and Presser, 2003; Nadler, Weston, and Voyles, 2014), indifference (Schaeffer and Presser, 2003; Nadler, Weston, and Voyles, 2014), uncertainty (Baka, Figgou, and Triga, 2012; Nadler, Weston, and Voyles, 2014), confusion, or to signal context dependence (e.g., “it depends” or disputing the question, see Baka, Figgou, and Triga, 2012).

The middle option may also be used for certain response styles, such as socially desirable responding (Sturgis, Roberts, and Smith, 2012) or satisficing (Krosnick, 1991), although there is not much research showing it actually leads to satisficing Wang & Krosnick (2020).

Af a middle alternative is explicitly offered, the proportion endorsing it increases dramatically (e.g. Ayidiya & McClendon, 1990; Bishop, 1987; Bishop, Hippler, Schwarz, & Strack, 1988; Kalton, Collins, & Brook, 1978; Kalton, Roberts, & Holt, 1980; Rugg & Cantril, 1944).

Some studies show that not including a middle option decreases validity and increases measurement error (O’Muircheartaigh, Krosnick, and Helic, 1999; Kahn, and Dhar, 2002)

Recent study on this: Wang & Krosnick (2020)

An alternative approach to this issue is to use branching. Respondents could first be asked whether they fall at the midpoint or on one side, followed by a question about their extremity on a side. This approach was found to be more reliable and valid than using a 7-point scale (Krosnick and Berent, 1993; Malhotra, Krosnick, and Thomas, 2009).

Conclusion: If it is possible that respondents may have a moderate view, it seems crucial for it to be possible to capture this view. Limitations of a middle option could then be addressed in other ways (e.g., clear questions).

Response Option Labeling

There are several studies that show all response options should be labelled, rather than only labeling the end points (Krosnick & Berent, 1993; Weng, 2004).

For an example of biopolar labels for a 2- to 11-point Likert scale, see Table 1.

It is also recommended to avoid agree-disagree response labels because asking respondents to rate their level of agreement is a cognitively demanding task that increases respondent error and reduces responding effort (Gehlbach & Brinkworth, 2011).

Possible labels, from CampusLabs:

Agreement: Strongly agree, Moderately agree, Neither agree nor disagree, Moderately disagree, Strongly disagree (another version removes the “moderately” qualifier and/or uses “neutral”)

Comparison: Much X, Slightly X, About the same, Slightly (opposite of X), Much (opposite of X)

Ease: Very easy, Moderately easy, Neither easy nor difficult, Moderately difficult, Very difficult

Expectations: Exceeds expectations, Fully meets expectations, Does not fully meet expectations, Does not meet expectations at all

Extent (5 pt): A great deal (Completely, if appropriate), Considerably, Moderately, Slightly, Not at all

Extent (4 pt): Significantly, Moderately, Slightly, Not at all

Frequency (no set time): Always, Often, Occasionally, Rarely, Never

Frequency (general): Daily, Weekly, Monthly, Once a semester, Once a year, Never

Frequency (based on time frame): More than 5 times, 4 - 5 times, 2 - 3 times, 1 time, Less than 1 time, Never

Frequency (extended): More than once a week, Once a week, Once a month, Once a semester, Once a year, Less than once a year, Never

Helpfulness: Extremely helpful, Very helpful, Moderately helpful, Slightly helpful, Not at all helpful

Importance: Extremely important, Very important, Moderately important, Slightly important, Not at all important

Interest: Extremely interested, Very interested, Moderately interested, Slightly interested, Not at all interested

Likelihood: Very likely, Moderately likely, Neither likely nor unlikely, Moderately unlikely, Very unlikely

Numeric Scales: Less than #, About the same, More than #

Probability: Definitely would, Probably would, Probably wouldn’t, Definitely wouldn’t

Proficiency: Beginner, Developing, Competent, Advanced, Expert (typical for Rubrics)

Quality: Excellent, Good, Average, Below average, Poor

Satisfaction: Very satisfied, Moderately satisfied, Neither satisfied nor dissatisfied, Moderately dissatisfied, Very dissatisfied (another version removes the “moderately” qualifier and/or uses “neutral”)

Taken from https://baselinesupport.campuslabs.com/hc/en-us/articles/204305485-Recommended-Scales

References