Cognitive reflection is associated with greater truth discernment for COVID-19 headlines, less trust but greater use of formal information sources, and greater willingness to pay for masks among social media users in Pakistan

---

Implications

The uncertainty surrounding the COVID-19 pandemic has been fueled by the COVID-19 “infodemic.” This has been particularly challenging for developing countries where lower literacy levels and lack of prior exposure to technology may make people more susceptible to COVID-19 misinformation (Guess et al., 2020). In this work, we ask whether cognitive reflection (Frederick, 2005; Evans & Stanovich, 2013)—the ability to use reflection to override intuitions—is associated with belief in COVID-19 misinformation, trust in different sources for receiving credible COVID-19 information, and COVID-19 preventative behaviors. More specifically, considering social media users in Pakistan, we tested whether higher cognitive reflection—as measured by CRT scores—predicts greater truth discernment, greater use of and trust in formal information sources, and higher WTP for masks.¹Formal sources may be defined as those which are constituted in some regularized or legal manner in relation to the user (e.g., TV channels, newspapers, government sources, and international sources such as WHO) (Kaye, 1995).

Examining these relationships in Pakistan, a developing country with the fifth highest population in the world and a fast-growing social media user base, contributes to our global understanding of the role cognitive reflection plays in explaining susceptibility to COVID-19 misinformation, since existing research has largely focused on the United States and Europe. We focus on cognitive reflection and the cognitive reflection test (CRT) (Frederick, 2005) as the latter has gained considerable popularity due to its power to predict a diverse range of beliefs, attitudes, and behavior. For example, higher CRT scores have been associated with fewer paranormal beliefs (Pennycook et al., 2012), lower susceptibility to bias in deductive reasoning (Toplak et al., 2011), greater rejection of conspiracy claims (Swami et al., 2014), lower perceived accuracy of fake news (Pennycook & Rand, 2019a), lower likelihood of sharing misinformation on social media (e.g., Pennycook et al., 2020), and greater sharing of reliable news sources on Twitter (Mosleh et al., 2021). However, there is a dearth of empirical evidence about CRT’s power to predict belief in COVID-19 misinformation and preventative COVID-19 behaviors in the context of developing countries that generally have lower literacy rates and education levels. One notable exception is a study by Rosenzweig et al. (2021), which found CRT scores to be positively associated with discernment among a sample of Facebook users in Nigeria.

In this paper, we aim to shed light on the relationship between cognitive reflection and vulnerability to COVID-19 misinformation using a sample of 621 social media users in Pakistan drawn randomly from low- and middle-income areas in the city of Lahore. We conducted a telephone survey due to COVID-19 restrictions (as opposed to a field survey) and to ensure representation of low digital literacy populations, who may not be well represented on online survey platforms (Hargittai & Shaw, 2020). We examine the association of cognitive reflection with truth discernment, trust in information sources, and WTP for KN95 masks via a real-stakes game.

To measure cognitive reflection, we used the verbal CRT proposed by Sirota et al. (2020), as the traditional CRT questions are known to confound reflection ability—or the disposition to think analytically—with mathematical ability (e.g., Campitelli & Gerrans, 2014; Liberali et al., 2012; Mastrogiorgio, 2015; Pennycook & Ross, 2016). As a result, they are less useful among populations with low numeracy skills or low literacy levels. In our study, we considered a sample of low- and middle-income social media users in Pakistan with a median education level, which lies between primary school and high school level education, and thus numeracy skills were likely to be low. More specifically, we used a 2-item CRT in our study adapted from the validated set of verbal CRT questions by Sirota et al. (2020).

To understand the relationship between cognitive reflection, beliefs about misinformation, and actual behavior, as well as to help address concerns that survey responses may not reflect real-world behavior, respondents participated in an incentive-compatible game—a variant of the Becker–DeGroot–Marschak (BDM) game (Becker et al., 1964)—designed to measure actual WTP involving relatively high-stakes. In this game, participants first chose a price at which they were willing to buy a KN95 mask in a given range of prices from PKR 0 to PKR 250 (USD 0 to USD 1.5 in 2020). A computer then randomly selected a price in the same range. If the computer’s price was lower than the participant’s bid, they get the KN95 mask, otherwise they receive money equal to the computer’s price.²We used easyload to transfer the received amount in airtime to the mobile phone of respondents. https://www.telenor.com.pk/easyload/ Thus, if a participant’s WTP for the mask was greater (e.g., closer to PKR 250), they are more likely to win the mask. Moreover, a positive correlation between CRT scores and WTP would indicate that individuals with higher CRT scores were more likely to have accurate risk perceptions about COVID-19 and thus have a greater WTP for masks.

We found that cognitive reflection is associated with greater truth discernment (i.e., the ability to distinguish between true and false headlines),³This measure of truth discernment has been used in several earlier works (e.g., Pennycook et al., 2019a; Pennycook et al., 2019b; Guess et al., 2020; Sirlin et al. 2021; and Rosenzweig et al., 2021). greater use but less trust in formal information sources (e.g., WHO, newspapers, government sources such as https://covid.gov.pk/). In addition, we find that those with higher CRT scores are more likely to bid a higher WTP for a KN95 mask in a real-stakes game. The positive association we observed between CRT scores and truth discernment is consistent with existing research (Pennycook & Rand, 2019a). Our work extends insights from existing research in two aspects: (i) we considered a low- and middle-income sample from a developing country with lower literacy levels than U.S. and European samples considered in earlier works, and (ii) we studied behavior—rather than just self-reported beliefs—through a real-stakes game, which showed that CRT scores predict demand for KN95 masks and thus contribute positively to COVID-19 preventative behaviors.

Our results emphasize the relationship between cognitive reflection and susceptibility to misinformation in the context of social media users in developing countries. Our research supports the “reflectionist” account of cognitive reflection (Pennycook, 2018) which emphasizes the importance of analytic thinking as a positive force in judgment and decision-making. And more specifically, individuals who are more willing to think analytically when given reasoning problems are less likely to fall for false news. To uncover possible underlying mechanisms, our multivariate regression analysis involving different outcome variables, including truth discernment and WTP for masks, shows that our results are not moderated by education, age, gender, or household expenditure;⁴The correlation between CRT and truth discernment as well as CRT and WTP for KN95 masks is robust to adding controls for age, gender, and household expenditure (statistically significant at the 5% level); see Appendix C. see Appendix C. The association between CRT and truth discernment can be explained, at least in part, by differences in deliberative reasoning between high and low CRT users. High CRT users are more likely to engage in deliberative reasoning, which has been found to be causally associated with better truth discernment (Bago et al., 2020).⁵In this study, the participants’ level of deliberation was experimentally manipulated, which demonstrated a causal effect whereby deliberation reduced belief in false (but not true) news, regardless of partisan alignment. Thus, it seems that people fail to distinguish between truth and falsehood partly because they do not stop to reflect sufficiently on their prior knowledge (or have insufficient or inaccurate prior knowledge). One possible mechanism driving COVID-19 preventative behaviors (e.g., demand for KN95 masks) could be that since higher CRT users exhibit greater truth discernment, they have more accurate beliefs about the COVID-19 pandemic compared to low CRT users. This suggests that they are also likely to have more accurate risk perceptions about the expected cost of contracting COVID-19, thereby affecting demand for KN95 masks. Another possible channel could be differences in the risk preferences of high and low CRT users. Frederick (2005) showed greater risk aversion among high CRT scorers for choices involving losses. In case of COVID-19, this translates into willingness to pay more for a KN95 (and the possible discomfort of wearing a mask) compared to the expected loss of suffering from illness or even death due to the COVID-19 infection.

Our findings have implications for social media platforms, governments, and citizens interested in countering COVID-19 misinformation by providing insights into which users are most vulnerable to misinformation. Our results suggest that interventions designed to counter COVID-19 misinformation should target low CRT individuals, who are worse at discerning true from false COVID-19 information. Moreover, our study highlights the importance of designing interventions to prompt individuals to engage in more reflection (Milkman et al., 2009). One such intervention could be the use of metacognitive reflection prompts (Salovich & Rapp, 2021), which involve asking questions that encourage people to reflect on their own susceptibility to inaccurate information (e.g., “When was the last time you remember relying on inaccurate information while reading?”). Such prompts have been experimentally shown to increase resistance to inaccurate information. An important benefit of metacognitive reflection is that it may motivate a more careful consideration of the accuracy of information and may also increase awareness of potential consequences of inaccurate information. Similarly, asking participants to pause and explain why a news headline is true or false reduces their intentions to share false information (Fazio, 2020). Recent work shows that there are several light-weight implementations of such prompts⁶These prompts included (i) asking participants how important it was to them to share only accurate news, (ii) providing digital literacy tips and (iii) asking people to judge the accuracy of a news headline.—designed to encourage people to reflect on the accuracy of presented information⁷Recent works show that individuals with higher CRT scores experience larger effects of light-weight accuracy prompt interventions, which also improved discernment even for participants who answered all CRT questions incorrectly (Epstein et al., 2021; Pennycook et al., 2021; Pennycook & Rand, 2022). Thus, Pennycook et al. (2022) hypothesize that accuracy prompts may in part work by inducing more reflection. Alternatively, they might just be changing factors that people reflect on without inducing more deliberation. However, further research is needed to distinguish between these cases.—which are effective and can be delivered at scale (Epstein et al., 2021). There are several ways in which social media companies, governments, and civil society organizations could deliver such interventions intended to encourage information evaluation, for example through social media ads, by showing a reflection prompt that appears when one tries to share content on a social media platform,⁸Social media companies already use prompting for different purposes, for example, Twitter prompts users when they try to share a tweet that was labeled for violating one of their policies. https://help.twitter.com/en/resources/addressing-misleading-info via public service announcements, etc. Moreover, researchers have called for incorporating reflection-based approaches (e.g., those involving metacognition) into media literacy education for countering exposure to misinformation (Hodgin & Kahne, 2018; Vraga et al., 2020). Finally, reflection prompts also have the normative benefit that they leverage an individual’s own, often latent, ability to evaluate information rather than relying on external arbiters (e.g., social media organizations, fact-checking organizations) to discern between truth and falsehood.

We hope that our work will help encourage efforts on using CRT to identify populations who may be more vulnerable to misinformation as well as in designing interventions that promote greater cognitive reflection to reduce the spread of misinformation online.

Findings

Finding 1: Users with higher CRT scores show greater truth discernment.

High CRT scores are positively associated with truth discernment—which is the ability of respondents to distinguish between true news and false news stories (defined as average accuracy ratings for true news minus average accurate ratings for false news)—for COVID-19 news stories. We found a significant positive relationship of CRT scores with truth discernment (ß = 0.139, 95% CI = [0.092, 0.185], p < 0.001); see Figure 1. These results are robust to adding controls for age, gender, education level (primary, up to high school, college education or above), and household expenditure (which we use as a proxy for income); see Appendix C. These findings are consistent with existing research, which found that higher CRT scores are correlated with ​​lower perceived accuracy of non-COVID fake news on a sample of U.S. adults recruited online (e.g., Lazer et al., 2018; Pennycook & Rand, 2019a) and COVID-19 misinformation among a sample of social media users in the United States (e.g., Pennycook et al., 2020) and Nigeria (Rosenzweig et al., 2021).

Next, we unpack whether truth discernment is driven by greater perceived accuracy of true news or lower perceived accuracy of false news by considering accuracy for false news and true news separately. We find that truth discernment is primarily driven by lower perceived accuracy of false news by respondents with higher CRT scores (ß = -0.132, 95% CI = [-0.172, -0.091], p < 0.001); see Figure 1. We find no significant relationship of CRT scores with the perceived accuracy of true news (ß = 0.0071, 95% CI = [-0.172, -0.091], p = 0.453); see Figure 1.⁹One possible reason for the high accuracy of true news could be that when we conducted the phone survey (i.e., in September 2020), the infection rates had declined considerably in Pakistan from the peak daily rate of around 7000 cases to 500 daily cases (see https://covid.gov.pk/stats/pakistan) and that people had formed accurate beliefs about the true news we asked them, which were related to handwashing, wearing a mask, and physical distancing. Thus, it is likely that the correlation between CRT scores and truth discernment is underestimated due to ceiling effects for true news even though we expect mean discernment to be higher for CRT groups without ceiling effects.

Finding 2: Higher CRT scores are associated with less trust but greater use of formal information sources. 

Examining trust of respondents on different sources, we found that high CRT scores are negatively associated with trust and confidence in formal information sources, which includes TV news channels, Urdu and English newspapers, radio, World Health Organization (WHO), and the government (ß = -0.343, 95% CI = [-0.508, -0.178], p < 0.001); see Figure 2.¹⁰These results are robust to the inclusion of controls for age, education, gender, and household expenditure; see Appendix Figure A. However, high CRT users more frequently obtain COVID-19 information from formal information sources compared to low CRT users; see Appendix E, Figure D. These results suggest that even though high CRT users more frequently obtain COVID-19 information, cognitive reflection might make users less trusting of content consumed from these sources.¹¹This is plausible given the COVID-19 “infodemic” and the uncertainty surrounding the COVID-19 pandemic especially in 2020. In comparison, for social media platforms (Facebook, WhatsApp, YouTube, and Twitter), we did not find a statistically significant association (at the 5% significance level) between CRT scores and trust (ß = 0.124, 95% CI = [-0.0094, 0.256], p = 0.067); see Figure 2. Moreover, this association is not robust to adding frequency of use of social media sources as a control (ß = -0.0139, 95% CI = [-0.143, 0.115], p = 0.832).

Is greater frequency of use of formal sources correlated with truth discernment? We found that greater use of formal sources is associated with better truth discernment (ß = 0.37, 95% CI = [0.158, 0.589], p = 0.001). We also found a positive association between frequency of use of social media platforms and truth discernment (ß = 0.32, 95% CI = [0.231, 0.409], p < 0.001). Thus, a potential channel through which higher CRT scores might be affecting discernment is through greater exposure and use of several sources to access information, thereby enabling users to form more accurate beliefs.

Finding 3: Outcomes of an incentive-compatible game show that users with higher CRT scores have greater willingness to pay for KN95 masks.

The outcomes of our real-stakes game designed to measure WTP show that higher CRT scores are positively associated with WTP for KN95 masks (ß = 21.7, 95% CI = [4.7, 38.8], p = 0.013); Figure 3. As a robustness check, we also report the proportion of KN95 masks won by respondents with different CRT scores. We find that these results are consistent with the WTP results. Taken together these findings suggest that respondents with higher CRT scores are more likely to have accurate beliefs about COVID-19 and therefore more accurate perception of the risks posed by COVID-19, resulting in greater willingness to pay for KN95 masks.

These results are also consistent with existing evidence relating cognitive reflection with actual behavior. For example, considering a sample of mostly UK- and US-based individuals, Mosleh et al. (2021) found that Twitter users with higher CRT scores are more discerning in their social media use, as evidenced by the types and number of accounts followed, and by the reliability of the news sources they shared. An intervention designed to nudge people to think about accuracy increased actual sharing discernment in a Twitter field experiment involving American individuals (Pennycook et al., 2021).

Methods

From September 1, 2020, to September 21, 2020, we conducted a phone survey of social media users chosen randomly based on population density in the city of Lahore,¹²We use population density as a proxy for income levels. We obtained contact information and consent to participate in follow-up surveys of a field sample recruited in an earlier study. See Appendix E for details about sample selection. the capital of the Punjab province in Pakistan, to understand the relationship between cognitive reflection and truth discernment of COVID-19 headlines, trust in social media platforms for receiving credible COVID-19 information, and COVID-19 preventative behaviors. Our final dataset included a total of 621 individuals: mean age = 29.3, 50.4% female, mean monthly household expenditure = PKR 35,575 (USD 214.5 in 2020), and a median education between primary school and high school.

Our survey included questions about demographic characteristics, social media use, and COVID-19 information sources and attitudes, and CRT questions. In addition, we asked a series of questions about 10 COVID-19 news stories (seven false and three true); see Table 1 in Appendix A for the list of headlines used. We obtained the true stories from major news outlets in Pakistan including DAWN and The Express Tribune as well as the World Health Organization (WHO). We used false COVID-19 news stories that had appeared online but were subsequently fact-checked and verified as untrue by various organizations including AFP Fact Check, PolitiFact, WHO, and UNICEF. Finally, the respondents participated in a real-stakes mask game. While all participants were presented with the same set of news stories, they were presented in a random order to each participant. After hearing each news story, we asked if they thought the news was true or not (“To what an extent do you agree with the statement?”) to which the participants responded using a slider scale 0 (disagree), 0.5 (not sure), and 1 (agree).¹³An important distinction between our phone survey and earlier studies that used online media formats to present false and true news is that the latter includes contextual information, such as the source of news stories whereas in our phone survey, no source information was provided. Thus, our survey measures the accuracy of already held beliefs about COVID-19 rather than their ability to use contextual clues (e.g., source) to determine the veracity of information. Therefore, our findings may generalize to situations where exposure to (mis)information might be high, and people have formed some beliefs about the accuracy of various statements. We then found respondent-level truth discernment, which is defined as average accuracy ratings for true news minus average accurate ratings for false news and is a measure of respondents’ ability to distinguish between true news and false news stories. The mean discernment across all respondents was 0.79.

Traditional CRT questions (Frederick, 2005) are known to confound reflection ability with mathematical ability and thus they are likely to be less useful for populations with low numeracy skills. Given the relatively low education level in our sample, we used a 2-item CRT from the validated set of verbal CRT questions by Sirota et al. (2020).¹⁴Earlier works have used CRTs comprising a different number of questions including a 3-item (Frederick 2005), a 4-item (Thomson et al., 2016), and a 7-item (Pennycook et al., 2019a) CRT. Each question had an obvious but incorrect response to assess individuals’ ability to suppress a default (intuitive) incorrect response in favor of a correct reflective response. We asked both questions in Urdu, the national language of Pakistan. The two CRT questions taken from Sirota et al. (2020) were as follows:

“If you were running a race, and you passed the person in second place, what place would you be in now?”. The correct answer is “second” whereas the intuitive answer is “first.”

“Fatima’s father has 4 daughters but no sons—Tina, Mina, Rina. What is the fourth daughter’s name?”¹⁵We changed the names in the original question from (Mary, Nana, Nene, Nini, Nono) to (Fatima, Tina, Mina, Rina) but retained the structure to reflect greater awareness of latter names in the local context. The correct answer is “Fatima” whereas the intuitive answer could be any name ending with the “ina” sound (e.g., Bina, Sina).

The CRT score was coded as follows: 0 (neither correct), 0.5 (one correct), 1 (both correct). One criticism of the traditional CRT questions has been their high familiarity (Sirota et al., 2020). In our sample, 88% of the respondents had not heard of the verbal CRT questions we asked. We found that our results are robust after controlling for a binary variable capturing whether a respondent was already aware of the CRT questions (1 if aware of both the questions, 0 otherwise).

We measured trust in social media platforms using the question, “How much trust and confidence do you have in news about the coronavirus received through the following social media platforms: WhatsApp, Facebook, Twitter, and YouTube.” For each platform, the response options were (1) Very low trust, (2) Low trust, (3) High trust, (4) Very high trust, (N/A) Do not use this source. We constructed a respondent-level trust variable by taking the average across all four platforms. The mean trust across all respondents was 2.78. We did not assign any value to the response Do not use this source. Similarly, we measured trust in formal information sources using the question, “How much trust and confidence do you have in each of the following sources when it comes to reporting about the coronavirus?: News Channels, Urdu Newspapers, English Newspapers, Government Sources (e.g., covid.gov.pk, press briefings, etc), Radio, the World Health Organization (WHO)” and reported the average trust across all these sources (the response options were the same as above) across CRT scores. The mean trust for formal sources across all respondents was 2.83.

To understand the relationship between cognitive reflection and actual behavior, respondents participated in an incentive-compatible game designed to measure WTP involving real stakes. We measured WTP for KN95 masks at the start of the game using the question, “Please choose a price at which you are willing to buy a KN95 mask in the range of PKR 0 – PKR 250.”¹⁶In 2020, PKR 250 was equal to USD 1.5. The survey participation compensation was PKR 100 (USD 0.6), which was separate from the money involved in the game. After choosing a price, a computer randomly selected a price in the same range. For ease of delivering money, the computer chose a price from a discrete but uniformly spaced set (0, 25, 50, 75, 100, 125, 150, 175, 200, 225, 250). If the computer’s price was lower than the participant’s bid, they received a KN95 mask,¹⁷The KN95 masks were delivered to the homes of the survey takers by our implementation partner. otherwise, they received an amount equal to the computer’s price as mobile airtime. Thus, if a participant’s WTP for the mask was greater (e.g., closer to PKR 250) then they were more likely to win the mask. In the text, we have reported the regression coefficient, p-value, and 95% confidence interval of the estimated coefficient from a regression of the outcome on the CRT score.