Article Text
PDF
Abstract
Objectives To develop and validate the Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C), a multidimensional scale that addresses seven domains over 20 items including stress, fear, loneliness, sense of community, belief that the pandemic is a hoax or exaggerated, the use of and attitude to non-pharmaceutical interventions and vaccine hesitancy, in a single measure.
Design Cross-sectional validation study.
Setting Internet based with respondents in the USA and UK.
Participants General community respondents using the Prolific Academic platform.
Main outcome measures Exploratory factor analyses with promax oblique rotation and confirmatory factor analysis including goodness of fit indices: root mean square error of approximation (RMSEA), standardised root mean square residual (SRMR) and comparative fit index (CFI). Reliability as internal consistency using Cronbach’s alpha. Convergent and discriminant validity using Pearson correlation coefficients.
Results The sample included 351 respondents in the USA and the factorial structure was confirmed using a separate set of 348 respondents in the UK. The OPAS-C had excellent goodness of fit characteristics, with an RMSEA of 0.047 (90% CI 0.037 to 0.056), SRMR of 0.043 and CFI of 0.962. Reliability was excellent, demonstrating Cronbach’s alpha of 0.87 in both the US and UK samples. Convergent validity showed correlation coefficients of 0.54 and 0.49 in the US and UK samples, respectively. Discriminant validity demonstrated correlations of 0.21 and 0.26 in the US and UK samples, respectively.
Conclusions The OPAS-C represents the first validated scale that addresses mental health measures and public health-relevant responses to COVID-19, and may be a useful measure for use in future longitudinal and cross-sectional studies. Further international validation beyond the USA and UK may be helpful.
- COVID-19
- public health
- epidemiology
Data availability statement
Data are available upon reasonable request. Data are available from the corresponding author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Inclusion of a range of stakeholders, including the general public, in the creation of the Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C) adds to the clinical meaningfulness and validity of the scale.
Encompassing a wide range of domains relevant to medical and public health researchers in a single scale may permit feasible, and therefore cost-effective, longitudinal and cross-cultural assessments.
The excellent fit characteristics of the factorial structure and the robustness of these findings across populations mean that the OPAS-C is generalisable across the populations studied.
Internet-based sampling may affect generalisability, and biases—such as response bias and social desirability bias—may have affected our findings.
Determining the sensitivity to change and test–retest reliability of the OPAS-C may be helpful in the future.
Introduction
Pandemic COVID-19 has taken a significant toll on the lives, liberties and livelihoods of people across the world, and understanding public attitudes to the disease is a high priority for public health researchers and medical and mental health experts.1–3 Public attitudes to COVID-19 and the responses of the general public to the pandemic, social distancing and other mitigation measures, and other restrictions may have a direct bearing on the success of public health efforts.4–6
While numerous studies have individually evaluated public attitudes to the COVID-19 pandemic,1 7–13 in order to better compare these attitudes, actions and beliefs over time and across geographic locations, validated scales provide greater insight and allow for more meaningful longitudinal and geographic cross-sectional comparisons.14 One of the focuses of the pandemic has been on the geographic heterogeneity in general public attitudes and behaviours across countries—and this variability has sometimes been associated with differential outcomes.15 Several scales have already been developed for selected public responses to COVID-19, including those that focus on fear,16 17 anxiety,18 stress (including distress, depression, anxiety and perception of danger),19 fatalism,20 knowledge,21 shopping behaviour22 and hope.23 While these scales are all important tools for researchers and clinicians, none of them were designed for public health researchers to address the attitudes and behaviours, positive and negative emotions and perceptions of vaccines and community that directly affect global health considerations.
We therefore aimed to develop and validate a comprehensive multidimensional scale that would assess the general public’s response to COVID-19 in its myriad dimensions, from fear, stress and loneliness through to positive responses such as sense of community, while also addressing key behavioural considerations that are vitally important to public health efforts, such as adherence and attitude to non-pharmaceutical interventions (NPIs) and vaccine hesitancy. Given the dynamic nature of the pandemic, the goal was to develop a short feasible scale so that it does not place an undue burden on respondents or researchers.
Methods
Item generation
The Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C) was developed following established psychometric protocols and best practices.14 Domain generation was performed via an iterative deductive and inductive approach. After a literature review, online focus groups with members of the public were used to develop a set of possible domains of interest. A group of expert reviewers in the fields of medicine and public health were also consulted regarding domain inclusion in keeping with standard scale development methodology.14 Domains of interest included (1) acute stress; (2) fear of contracting COVID-19; (3) anxiety; (4) loneliness; (5) use of and attitude to NPIs; (6) belief that the pandemic is a hoax or exaggerated; (7) sense of community and optimism; and (8) vaccine hesitancy. All responses were recorded on a 5-point Likert scale ranging from strongly agree to strongly disagree. Items were reverse scored, as appropriate, to yield a scale where higher values represented greater levels of concern regarding COVID-19. All subjects provided informed consent for participation and were permitted to withdraw from their anonymous surveys at any time. Statistical analyses were performed using Stata for Mac V.13 (Stata, College Station, Texas, USA) and Lavaan V.0.6-8 (University of Ghent, Belgium).
Data collection
Two separate samples were used in the development of the OPAS-C. First, a sample of 350 respondents in the USA were recruited through Prolific Academic (Oxford, UK), an academic survey panel site, and a draft scale was presented using the Qualtrics platform (Qualtrics, Provo, Utah, USA). All respondents were rewarded with a small payment (<£1) and were offered the possibility of withdrawing their consent for anonymous participation at any time. This sample was used to develop the scale and perform exploratory factor analysis, as detailed below. A second sample of respondents from the UK was then solicited in a similar fashion using the same electronic survey recruitment tools. This sample was used for confirmatory factor analysis. Results from the USA and UK were evaluated separately to establish the robustness of the scale. Demographic information was self-reported by respondents and was linked with responses through an anonymous 25-digit alphanumeric key. Baseline information included age, sex, employment status, education history and household income.
Domain and item reduction and exploratory factor analysis
Maximum likelihood parallel analysis was performed to help determine the appropriate number of domains to retain for the overall scale.24 This approach is more conservative than the reliance on eigenvalues over 1, and a recent study has highlighted the importance of relying on more than one approach when deciding on the number of factors to retain.25 26 Item reduction was approached by examining polychoric correlations (item–item correlations) and polyserial correlations (item–test correlations). Though low correlations (<0.3) may suggest consideration of deletion, we decided a priori that since the broadly multidimensional scale includes domains that would be expected to correlate poorly, then the potential domain-relevant contributions of the item may supersede the absolute correlation values.27 Exploratory factor analysis was performed using maximum likelihood estimation followed by oblique factor rotation.
Confirmatory factor analysis
The robustness of the findings from the US sample was tested by performing confirmatory maximum likelihood factor analysis using structural equation modelling on the UK sample with the model developed in the US sample using maximum likelihood estimation with robust SEs and a Satorra-Bentler scaled test statistic. Goodness of fit indices including the root mean square error of approximation (RMSEA), the standardised root mean square residual (SRMR) and the comparative fit index (CFI) were assessed to gauge the model fit. Values of RMSEA ≤0.06, SRMR ≤0.08 and CFI ≥0.95 suggest an excellent fit.28
Reliability and validity
Reliability as internal consistency was assessed by examining Cronbach’s alpha for the OPAS-C. An ideal scale has high internal consistency, though when correlations are very high, this may suggest the possibility of redundancy; therefore, the ideal range for Cronbach’s alpha is between 0.75 and 0.9.29
Convergent validity was assessed by examining the correlation between the OPAS-C and agreement with a single question addressing the respondents’ overall response to the pandemic (‘thinking about the pandemic makes me feel anxious’) in both the US and UK samples. Discriminant validity was assessed by the correlation between the OPAS-C score and agreement with a non-specific statement (‘I think we are all in this together’) in the US and UK samples. A cut-off of 0.3 was used for these correlations, so that values >0.3 are desirable for convergent validity and values <0.3 are desirable for discriminant validity, consistent with prior studies.19
Patient and public involvement
This study was designed to develop a scale that could be used to measure public responses to COVID-19. As such, online anonymous focus groups were used to aid in domain generation and public responses were solicited through Prolific Academic to help develop the scale from the time of inception, and the goal of the research was to reflect the priorities and preferences of the public. Results will be disseminated and hopefully lead other researchers to use the OPAS-C in assessing public responses to the COVID-19 pandemic.
Results
Respondent characteristics
For the US sample, of the 359 respondents who started the survey, 351 subjects returned the finished surveys, yielding a 97.8% completion rate in the survey panel. The mean (SD) age was 30 (10.5), and respondents ranged in age from 18 to 72. 46.8% of respondents were male, and the median (IQR) household income was $50 000 (26 000, 95 000) for those who were working. A total of 26.3% of respondents were not currently working; 15.2% of respondents were high school graduates and 49.1% had university degrees.
In the UK sample, of the 354 respondents who started the survey, 348 subjects returned the finished surveys, yielding a 98.3% completion rate in the survey panel. The mean (SD) age was 30.5 (11.3), and respondents ranged in age from 18 to 71. 51.4% of respondents were male, and the median (IQR) household income was £29 550 (16 750, 48 500) for those who were working. A total of 20.6% of respondents were not currently working; 20.1% of respondents were high school graduates and 37.9% had university degrees.
Domain and item reduction and exploratory factor analysis
A total of 50 items were developed which were reduced to 43 prior to testing; the large item pool was designed to test multiple similar iterations of items in a focus group format. We included between 4 and 10 items per domain a priori for consideration, depending on the clinical breadth of the domain. Maximum likelihood parallel analysis suggested a 7-factor solution. A total of 23 items were deleted due to low polychoric and polyserial correlations when clinically relevant, unifactorial outcomes or redundancy. Factor loadings from the 20 retained items after maximum likelihood exploratory factor analysis and promax oblique rotation are presented in table 1. The eigenvalues for these factors were 3.97, 2.49, 1.69, 1.12, 1.40, 1.25 and 1.48. This 7-factor solution reflected a simple structure with appropriate loading. Factors correlated with the domains of (1) acute stress; (2) fear; (3) loneliness; (4) sense of community; (5) belief that the pandemic is a hoax or exaggerated; (6) use of and attitude to NPIs; and (7) vaccine hesitancy. Anxiety as a stand-alone domain was culled due to unifactorial loading and significant overlap with the more general stress domain. The final scale and scoring is presented in table 2 and factor intercorrelations are presented in table 3.
Factor loadings in the US sample exploratory factor analysis
The OPAS-C
Factor intercorrelations
Confirmatory factor analysis
Confirmatory factor analysis was first performed on the UK sample using the model developed in the US sample. Using maximum likelihood estimation with robust SEs and a Satorra-Bentler scaled test statistic, the 7-factor model including 20 items demonstrated excellent goodness of fit characteristics, with RMSEA=0.047 (90% CI 0.037 to 0.056), SRMR=0.043 and CFI=0.962.
The total OPAS-C scores were normally distributed in both the US and UK populations as demonstrated by the results from the Shapiro-Wilk test (p>Z=0.31 and 0.30 for the US and UK populations, respectively). Mean and 95% CI for the OPAS-C were 54.7 (53.5 to 56.0) in the US population and 56.1 (54.9 to 57.2) in the UK population. The similarities in OPAS-C distributions between the two populations further bolster the robustness of the scale.
Reliability and validity
Cronbach’s alpha for the overall OPAS-C was 0.87 in both the US and UK samples, suggesting excellent reliability as measured by internal consistency in both populations. Reliability for each subscale ranged from 0.66 to 0.93 in the US sample, as outlined in table 4. Convergent validity, as assessed by the correlation between the OPAS-C and agreement with a single question addressing the overall response to the pandemic (‘thinking about the pandemic makes me feel anxious’), was good at 0.54 and 0.49 in the US and UK samples, respectively. Discriminant validity, as assessed by the correlation between the OPAS-C score and agreement with a non-specific statement (‘I think we are all in this together’), was also good, with values of 0.21 and 0.26 in the US and UK samples, respectively.
Subscale reliability, as measured by Cronbach’s alpha
Discussion
We found that the OPAS-C, the first comprehensive validated multidimensional scale assessing public attitudes and responses to the COVID-19 pandemic, was robust across both US and UK populations and performed well on goodness of fit indices as well as in reliability and validity testing. The robustness of the factorial structure of the scale, coupled with its excellent fit and broad domain structure, as well as its outstanding feasibility, suggests that the OPAS-C may represent a useful addition in the public health response to COVID-19.
Several other scales have been previously validated to assess public responses to the COVID-19 pandemic, but most of these are either unidimensional or address only psychological responses. Therefore, while these scales are excellent for their intended purpose, our interest was in globally evaluating attitudes, actions and beliefs and developing a single measure that may be useful across populations and over different timeframes for assessing the public health burden and response to COVID-19. The OPAS-C does not claim to be superior to any other validated scale; the aim of the scale, however, and its scope, are different from those that have been previously developed in that it is expressly designed to include the key information that public health researchers need—while also perhaps eliding over some of the details that would be important to those studying only mental health outcomes. Given the burden that the COVID-19 pandemic has already placed on health and economic systems worldwide,30 there is a clear need for a scale that can be easily deployed and is feasible (the 20-item OPAS-C can be completed in under 3 min) and that can capture myriad responses to the pandemic.
Additional strengths of this validation study include the use of both US and UK populations for the validation process and the broad inclusion of societal responses, from stress and fear to sense of community and vaccine hesitancy. Including various stakeholders, such as the general public, as well as researchers, is an important aspect of scale development, and helps ensure that the scale has adequate content validity.31 32 Moreover, the excellent fit characteristics of our factorial structure and the robustness of these findings across populations suggest that the OPAS-C may be a valuable adjunct for public health and social science researchers alike.
There are several limitations to this work. First, the sampling used in developing and validating the OPAS-C may not be representative of the overall population of interest. The robustness of the factorial structure of the scale and the outcomes overall—the US and UK samples differed by only 1.4 points out of a total possible score of 100—suggests that the generalisability of the OPAS-C is likely adequate. Further validation in multiple countries and different languages may be helpful to better establish the generalisability of this scale beyond the USA and UK. Second, further convergent and discriminant validity assessments could be considered using other established scales; since the OPAS-C is unique in its broad multifactorial structure, however, this would require the inclusion of multiple scales and additional questions. Moreover, the convergent and discriminant validity seen using single-question comparators support the validity of the OPAS-C. Still, since these rely on single-question comparators this can be regarded as preliminary validity evidence. Third, the internal consistency reliability of some of the subscales was limited—particularly for the items regarding pandemic exaggeration—and this may be a function of both having only two items included in the subscale and the discordance between the potentially polarising word ‘hoax’ and more general concerns regarding whether concerns have been exaggerated; future research in this area may be warranted. Fourth, we did not evaluate the test–retest reliability of the OPAS-C or its sensitivity to change over time. Future work that addresses the sensitivity of the OPAS-C to change as attitudes and behaviours evolve will be helpful. Finally, as with any study that relies on online data collection, biases such as response bias and social desirability bias may affect the results, though the replicability and robustness of our findings across disparate populations suggest that the OPAS-C is a valid and reliable scale for gaining valuable public health insight into pandemic COVID-19, and may serve as a useful addition to the burgeoning array of valid and clinically meaningful scales that can be used in studying COVID-19.
Data availability statement
Data are available upon reasonable request. Data are available from the corresponding author.
Ethics statements
Ethics approval
This research was approved by the Ascension Health Institutional Review Board through exempted approval as no identifying information was obtained from anonymous participants (approval 1580959-1).
References
Footnotes
Contributors Study conception: JK. Study design: JK, BNK. Statistical analyses: JK, BNK. Writing: JK, BNK. Oversight: JK.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.