Data quality checks
Unfortunately, not all respondents pay careful attention to survey instructions and survey items. We therefore would like to be able to detect these participants and prevent them from participating in the survey or exclude them from data analysis.
Some survey platforms have rules about what kind of checks they allow. Prolific is one of them. Make sure to follow the link to check their rules so you don’t break them.
There are different types of checks to detect inattentive participants: attention checks, comprehension checks, and commitment checks.
Attention checks
Attention checks are defined as simple ways to determine who is paying attention to your study instructions, and the amount of effort participants devote to reading questions before answering them (Oppenheimer et al., 2009).
Instructional Manipulation Checks (IMCs)
Instructional manipulation checks (IMCs) explicitly instruct a participant to complete a task in a certain way, and are therefore designed to see whether or not a participant has paid attention to the question being asked.
IMCs can be used in different formats (e.g., multiple-choice, text-based). Each format has advantages and disadvantages. Multiple-choice formats can be answered correctly randomly but text-based answers may be more difficult to analyze.
Examples
Binary
- Please select ‘True’ for this question. [True, False]
Multiple-choice
The color test you are about to take part in is very simple, when asked for your favorite color you must select ‘Green’. This is an attention check. Based on the text you read above, what color have you been asked to enter? [Red, Blue, Green, Organce, Brown] (Prolific’s Attention and Comprehension Check Policy, 2024)
For this question, please select ‘Somewhat Agree’ as your answer. [Strongly disagree, Disagree, Somewhat disagree, Somewhat agree, Agree, Strongly agree]
This is a simple question. You don’t need to be a wine connoisseur or avid beer drinker to answer. When asked for your favorite drink, you need to select carrot juice. Based on the text above, what is your favorite drink? [Wine; Beer; Vodka; Whiskey; Carrot juice; Other]
Help us keep track of who is paying attention. Please select “Somewhat disagree” from the options below. [Multiple choice: Strongly agree - Strongly disagree] (Stagnaro et al., 2024)
Please ignore this question and do not answer. [choose all that apply: Strongly agree Strongly disagree] (Stagnaro et al., 2024)
Text-based
In the box below, please type the word ‘Survey’ to indicate you are paying attention.
Please enter the number ‘5’ in the box below.
The following is to verify that you are a real person. Please enter the word “purple” in the box below. (Geisen, 2022)
Please write “twenty-five” using numbers. (Stagnaro et al., 2024)
Slider
- Please position the slider to the number 5.
Nonsensical Items
Nonsensical item checks embed a nonsensical item within a survey, to which only one or two responses to the question’s options can be justified as objectively correct.
Examples
I swim across the Atlanic Ocean to get to work every day. [Strongly disagree, Disagree, Agree, Strongly agree] (Prolific’s Attention and Comprehension Check Policy, 2024)
I work fourteen months in a year. [Yes, this is true of me; No, this is not true of me]
I have never used a computer. [Yes, this is true of me; No, this is not true of me]
What type of social media accounts do you use (if any)? Please answer honestly. [do they select the option “TisFask” which does not exist nor sound like any social media option] (Stagnaro et al., 2024)
While watching the television, have you ever had a fatal heart attack? [6-point Likert ranging from “Never” to “Often”] (Paolacci et al., 2010)
Are you licensed to operate a class SSGN submarine? [Yes/No] (Kennedy, 2022)
Which of the following did you do in the past week? Check all that apply. [Purchased a private jet; Climbed a peak in the Karakoram Mountains; Watched TV; Learned to cook halusky; Played jai alai; Read a book; None of the above] (Kennedy, 2022)
Factual or logical questions
Factual or logical questions consist of asking a question with a correct answer that everyone is likely to know.
Examples
What is 2 + 2? [Text box]
Which of the following is a primary color? [Green, Orange, Blue]
If yesterday was Monday, what is today? [Text box]
Which of these is a vegetable? [Egg, Salmon, Broccoli, Cheeseburger, Pizza, Milk] (Geisen, 2022)
Which of the following is a fruit? [Cow, Elephant, Chair, Banana].
Which of the following words is most closely associated with ‘book’? [Pen; Saw; Car; Pants; House]
Reverse wording
Questions with reverse wording ask the same thing twice but in a different way such that a response to one has the opposite meaning of the response to the other. If they respondent is paying attention, they should provide roughly the same answer for the first question and the inverse of the second question.
Examples
- “I always order the most expensive item on the menu.” vs. “I never order the cheapest item on the menu.”
Comprehension checks
Comprehension checks are used to measure participant’s level of understanding of specific study instructions to ensure that they understand what is required of them
Commitment checks
Commitment checks can be used to measure whether participants are motivated to answer questions and answer them honestly.
Examples
- We care about the quality of our survey data. For us to get the most accurate measures of your opinions, it is important that you provide thoughtful answers to each question in this survey. Do you commit to providing thoughtful answers to the questions in this survey? [I can’t promise either way; Yes, I will; No, I will not] (Geisen, 2022)