When presenting a Likert-type scale, you must choose which end of the scale to present first. On a horizontal scale, should “Strongly agree” appear on the left or “Strongly disagree”? On a vertical scale, which appears at the top?
Two mechanisms have been proposed to explain why response order can affect answers.
Satisficing (Krosnick, 1991) — when questions are difficult or motivation is low, respondents may select the first acceptable option without fully evaluating the rest. In visual surveys this biases responses toward earlier options; in telephone surveys toward later ones, since the last option heard remains most accessible in memory.
Anchoring-and-adjustment (Yan & Keusch, 2015) — respondents anchor on the first option and adjust insufficiently as they work through the remaining alternatives, biasing responses toward the starting point of the scale.
Table 1: Overview of empirical studies on response order effects.
| Source | Topic | Scale | Mode | N | Sample | Main finding |
|---|---|---|---|---|---|---|
| Friedman et al. (1994) | Student attitudes toward college | 5-pt agree–disagree, fully labeled, horizontal | PAPI | 208 | US undergrads | Primacy effect: strongly-agree-first yields more agreement on favorably worded items |
| Weng & Cheng (2000) | Personal distress | 5-pt fully labeled, within-subjects | PAPI | 858 | Taiwanese students | No substantial response order effects on means or psychometric properties |
| Rammstedt & Krebs (2007) | Personality (BFI-10) | 7-pt endpoint-labeled | PAPI | 315 | German students | Scale direction alone has no effect; number–label directional mismatch does affect responses |
| Yan & Keusch (2015) | Country development ratings | 11-pt endpoint-labeled unipolar | CATI | 498 | US adults (RDD) | Primacy effect for high-rated countries; consistent with anchoring-and-adjustment |
| Malhotra (2008) | Various attitudes | Multi-item, horizontal | CAWI | 397 | US adults (KnowledgePanel) | Primacy effect strongest for fast, low-education respondents |
| Christian et al. (2009) | Various | Various, horizontal and vertical | CAWI | ~3,156 | US undergrads | Response order effects mainly for vertical scales; nonlinear layout creates position bias |
| J. Höhne & Lenzner (2015) | Various attitudes | 5-pt agree–disagree, horizontal and vertical | PAPI (+ eye-tracking) | 84 | German participants (GESIS) | Response order effects larger for vertical than horizontal; eye-tracking confirms skipping of terminal categories |
| Krebs & Hoffmeyer-Zlotnik (2010) | Job importance | 8-pt endpoint-labeled | PAPI | 436 | German students | Primacy effect: positive-first yields higher importance ratings; factor structure unaffected |
| Krebs & Bachner (2018) | Job characteristics | 7-pt endpoint-labeled unipolar | PAPI | — | German and Israeli students | Primacy effect present in both left-to-right and right-to-left reading-direction groups |
| Garbarski et al. (2015) | Self-rated health | 5-pt fully labeled (excellent–poor) | CAWI | 2,696 | US adults (KnowledgePanel) | Primacy effect: excellent-first order yields better self-rated health |
| Garbarski et al. (2016) (Study 1) | Self-rated health | 5-pt fully labeled (excellent–poor) | CAWI | 1,347 | US adults (KnowledgePanel, TESS) | Primacy effect: excellent-first yields better SRH |
| Garbarski et al. (2016) (Study 2) | Self-rated health | 5-pt fully labeled (excellent–poor) | CAWI | 499 | US adults (KnowledgePanel, RTI) | Primacy effect replicated |
| Keusch (2018) | Brand attribute ratings | 6-pt endpoint-labeled unipolar | CAWI | 502 | Austrian non-probability panel | Primacy effect for all respondents; stronger for speeders; consistent with satisficing and anchoring-and-adjustment |
| J. K. Höhne & Krebs (2018) | Various attitudes | 5-pt agree–disagree and item-specific | PAPI | 930 | German students | Scale direction effects found for agree–disagree but not item-specific question formats |
| Liu & Keusch (2017) | Various attitudes (ANES) | Multi-item attitude scale | CAWI, CAPI | — | US adults (ANES 2012) | Scale direction increases acquiescence in web mode but not face-to-face |
| Yan et al. (2018) (ANES 2008) | Various political attitudes | Multi-item, various formats | CAPI | 2,322 | US adults (ANES 2008) | Scale direction effects stronger for non-attitudinal items and longer scales |
| Yan et al. (2018) (ANES 2012) | Various political attitudes | Multi-item, various formats | CAWI, CAPI | 5,914 | US adults (ANES 2012) | Effects stronger for earlier items in battery; self-administered shows more effects than CAPI |
| Terentev & Maloshonok (2019) | Course satisfaction and learning | 5-pt vertical | CAWI | ~22,910 | MOOC learners | Primacy effect in vertical item-by-item format (especially low-education); no effect in grid format |
| Smyth et al. (2019) | Satisfaction | 4-pt fully labeled, horizontal | CAWI, Mail | — | US adults, multiple surveys | Positive-first ordering yields consistently higher satisfaction ratings |
| Garbarski et al. (2019) | Self-rated health | 5-pt fully labeled (excellent–poor) | CAWI | 2,945 | US adults (MTurk) | Primacy effect robust across vertical, horizontal, and banked scale orientations |
| Krebs & Höhne (2021) | Various | Endpoint-labeled, vertical | CAWI | 3,401 | German non-probability access panel | Minimal scale direction effects; device type (PC vs. smartphone) does not moderate effects |
| Stefkovics (2022) | Political attitudes, life satisfaction | 5-pt agree–disagree, 11-pt numeric, 6-pt semantic differential | CAPI, CATI, CAWI | — | Hungarian adults | Few effects in interviewer-administered modes; scale direction effects emerge for numeric scale in online mode |
| Wöhner & Franzen (2024) | Life satisfaction | 11-pt numeric (0–10) | CAWI | 3,138 | Swiss adults | Descending scale yields ~0.7 pts lower satisfaction; ascending scale shows higher construct validity |
| Robie et al. (2022) | Personality (HEXACO) | 5-pt fully labeled | PAPI | 1,198 | Canadian students | No meaningful response order effects for incremental or decremental orderings |
For theoretical accounts of the mechanisms underlying these effects, see Krosnick (1991) (satisficing) and Schwarz et al. (1992) (cognitive elaboration model). Knauper (1999) is a meta-analysis of 14 Schuman & Presser telephone experiments showing that respondents aged 65+ are approximately twice as susceptible to response order effects as younger respondents, independent of education.
In self-administered surveys—whether CAWI or PAPI—a primacy effect is reliably observed: the first response option in a sequence attracts disproportionate endorsement. For agree–disagree scales, placing “Strongly agree” first inflates agreement relative to placing “Strongly disagree” first. In telephone (CATI) surveys, recency effects are more common. The effect is amplified for speeders and less motivated respondents (Keusch, 2018; Malhotra, 2008), older respondents (65+) (Knauper, 1999), non-attitudinal items, and longer scales (Yan et al., 2018). Switching to item-specific response options substantially reduces susceptibility to scale direction effects (J. K. Höhne & Krebs, 2018) and also improves reliability and validity (Dykema et al., 2022; Saris et al., 2010). When agree–disagree scales are unavoidable, a negative-to-positive ordering (Disagree→Agree) is recommended for self-administered surveys to reduce the risk of inflated agreement (Wöhner & Franzen, 2024; Yonnie Chyung et al., 2018).
Christian, L. M., Parsons, N. L., & Dillman, D. A. (2009). Designing Scalar Questions for Web Surveys. Sociological Methods & Research, 37(3), 393–425. https://doi.org/10.1177/0049124108330004
Dykema, J., Schaeffer, N. C., Garbarski, D., Assad, N., & Blixt, S. (2022). Towards a reconsideration of the use of agree-disagree questions in measuring subjective evaluations. Research in Social and Administrative Pharmacy, 18(2), 2335–2344. https://doi.org/10.1016/j.sapharm.2021.06.014
Friedman, H. H., Herskovitz, P. J., & Pollack, S. (1994). The biasing effects of scale-checking styles on response to a Likert scale. Proceedings of the American Statistical Association Annual Conference: Survey Research Methods, 792–795.
Garbarski, D., Schaeffer, N. C., & Dykema, J. (2015). The effects of response option order and question order on self-rated health. Quality of Life Research, 24(6), 1443–1453. https://doi.org/10.1007/s11136-014-0861-y
Garbarski, D., Schaeffer, N. C., & Dykema, J. (2016). The effect of response option order on self-rated health: A replication study. Quality of Life Research, 25(8), 2117–2121. https://doi.org/10.1007/s11136-016-1249-y
Garbarski, D., Schaeffer, N. C., & Dykema, J. (2019). The Effects of Features of Survey Measurement on Self-Rated Health: Response Option Order and Scale Orientation. Applied Research in Quality of Life, 14(2), 545–560. https://doi.org/10.1007/s11482-018-9628-x
Höhne, J. K., & Krebs, D. (2018). Scale direction effects in agree/disagree and item-specific questions: A comparison of question formats. International Journal of Social Research Methodology, 21(1), 91–103. https://doi.org/10.1080/13645579.2017.1325566
Höhne, J., & Lenzner, T. (2015). Investigating response order effects in web surveys using eye tracking. Psihologija, 48(4), 361–377. https://doi.org/10.2298/PSI1504361H
Keusch, F. (2018). Is satisficing responsible for response order effects in rating scale question? Survey Research Methods, Vol 12, 259–270 Pages. https://doi.org/10.18148/SRM/2018.V12I3.7263
Knauper, B. (1999). The Impact of Age and Education on Response Order Effects in Attitude Measurement. Public Opinion Quarterly, 63(3), 347. https://doi.org/10.1086/297724
Krebs, D., & Bachner, Y. G. (2018). Effects of Rating Scale Direction under the Condition of Different Reading Direction. Methods, data, 21 Pages. https://doi.org/10.12758/MDA.2017.08
Krebs, D., & Hoffmeyer-Zlotnik, J. H. P. (2010). Positive First or Negative First?: Effects of the Order of Answering Categories on Response Behavior. Methodology, 6(3), 118–127. https://doi.org/10.1027/1614-2241/a000013
Krebs, D., & Höhne, J. K. (2021). Exploring Scale Direction Effects and Response Behavior across PC and Smartphone Surveys. Journal of Survey Statistics and Methodology, 9(3), 477–495. https://doi.org/10.1093/jssam/smz058
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305
Liu, M., & Keusch, F. (2017). Effects of scale direction on response style of ordinal rating scales. Journal of Official Statistics, 33(1), 137–154. https://doi.org/10.1515/jos-2017-0008
Malhotra, N. (2008). Completion Time and Response Order Effects in Web Surveys. Public Opinion Quarterly, 72(5), 914–934. https://doi.org/10.1093/poq/nfn050
Rammstedt, B., & Krebs, D. (2007). Does Response Scale Format Affect the Answering of Personality Scales? Assessing the Big Five Dimensions of Personality with Different Response Scales in a Dependent Sample. European Journal of Psychological Assessment, 23(1), 32–38. https://doi.org/10.1027/1015-5759.23.1.32
Robie, C., Meade, A. W., Risavy, S. D., & Rasheed, S. (2022). Effects of Response Option Order on Likert-Type Psychometric Properties and Reactions. Educational and Psychological Measurement, 82(6), 1107–1129. https://doi.org/10.1177/00131644211069406
Saris, W., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options. Survey Research Methods, Vol 4, 61–79 Pages. https://doi.org/10.18148/SRM/2010.V4I1.2682
Schwarz, N., Hippler, H.-J., & Noelle-Neumann, E. (1992). A Cognitive Model of Response-Order Effects in Survey Measurement. In N. Schwarz & S. Sudman (Eds.), Context Effects in Social and Psychological Research (pp. 187–201). Springer New York. https://doi.org/10.1007/978-1-4612-2848-6_13
Smyth, J. D., Israel, G. D., Newberry, M. G., & Hull, R. G. (2019). Effects of Stem and Response Order on Response Patterns in Satisfaction Ratings. Field Methods, 31(3), 260–276. https://doi.org/10.1177/1525822X19860648
Stefkovics, Á. (2022). Are Scale Direction Effects the Same in Different Survey Modes? Comparison of a Face-to-Face, a Telephone, and an Online Survey Experiment. Field Methods, 34(3), 206–222. https://doi.org/10.1177/1525822X221105940
Terentev, E., & Maloshonok, N. (2019). The impact of response options ordering on respondents’ answers to rating questions: Results of two experiments. International Journal of Social Research Methodology, 22(2), 179–198. https://doi.org/10.1080/13645579.2018.1510660
Weng, L.-J., & Cheng, C.-P. (2000). Effects of Response Order on Likert-Type Scales. Educational and Psychological Measurement, 60(6), 908–924. https://doi.org/10.1177/00131640021970989
Wöhner, F., & Franzen, A. (2024). The Impact of Answer Scale Orientation on the Measurement of Life Satisfaction. Journal of Happiness Studies, 25(6), 78. https://doi.org/10.1007/s10902-024-00798-9
Yan, T., & Keusch, F. (2015). The effects of the direction of rating scales on survey responses in a telephone survey. Public Opinion Quarterly, 79(1), 145–165. https://doi.org/10.1093/poq/nfu062
Yan, T., Keusch, F., & He, L. (2018). The Impact of Question and Scale Characteristics on Scale Direction Effects. Survey Practice, 11(2), 1–10. https://doi.org/10.29115/SP-2018-0008
Yonnie Chyung, S. Y., Kennedy, M., & Campbell, I. (2018). Evidence-Based Survey Design: The Use of Ascending or Descending Order of Likert-Type Response Options. Performance Improvement, 57(9), 9–16. https://doi.org/10.1002/pfi.21800