Next Article in Journal
Eating Patterns in Children with Autism Spectrum Disorder
Previous Article in Journal
Prevalence of Sexual Dysfunction in Mexican Women with Rheumatoid Arthritis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Development and Validation of a Questionnaire to Assess Knowledge and Attitudes toward COVID-19 Preventive Measures in Romania

1
Faculty of Medicine, University of Medicine and Pharmacy “Grigore T Popa”, 700115 Iasi, Romania
2
Diabetes, Nutrition and Metabolic Disease Clinic, 700111 Iasi, Romania
3
Clinic of Pulmonary Diseases, 700115 Iasi, Romania
4
Institute of Cardiovascular Diseases “Prof. Dr. George I.M. Georgescu”, 700503 Iasi, Romania
*
Authors to whom correspondence should be addressed.
Healthcare 2022, 10(10), 1827; https://doi.org/10.3390/healthcare10101827
Submission received: 5 August 2022 / Revised: 19 September 2022 / Accepted: 19 September 2022 / Published: 21 September 2022

Abstract

:
The World Health Organization warns about the threat of the COVID-19 sixth wave. Our aim was to propose the first validated Romanian questionnaire to assess people’s level of education and attitudes towards general measures to protect against COVID-19 infection. Our study was conducted on a sample of 194 people. The first version of the questionnaire consisted of 40 items. Items that did not meet psychometric criteria were removed. Latent components/factors were identified through exploratory factorial analysis (EFA). The Cronbach’s alpha coefficient was used to assess internal fidelity. The EFA identified three factors. Factor 1 was named “Compliance with protective measures”, factor 2 was “Attitudes toward vaccination” and factor 3 was “Attitudes regarding potential COVID-19 therapies”. The final version of the questionnaire consists of 16 items. The test’s final score predicted the presence of vaccination with an accuracy of 0.773. The questionnaire score, the diagnosis of diabetes, the advice provided by healthcare workers and the medical profession proved to be significant predictors of vaccination. The implementation of our questionnaire within national programs could identify populational areas that need specific interventions to reach vaccination targets and prevent a full-blown sixth wave of the COVID-19 pandemic in Romania.

1. Introduction

The global COVID pandemic has affected more than half a billion people and has so far led to around 6,332,618 deaths, driving economic, political and social changes. The protective measures against SARS-CoV-2 virus infection are currently both general and recommended in all epidemics with respiratory-transmitted pathogens (social distancing, wearing protective masks), but also specific such as testing and vaccination. The role of these measures is to reduce the number of severe forms of disease and death but also to prevent the overcapacity of hospitalization and intensive care at the national level [1].
In a time of measures’ relaxation (the removal of the mandatory wearing of the mask in public places, the changes to the rules for testing and isolating new cases) after a period of decline in the incidence of COVID-19 cases throughout March, April and May 2022, our paper provides a warning signal based on latest World Health Organization (WHO) briefing (6 July 2022) that urges countries to readopt tried and tested public health measures and plug immunity gaps as infections new sub-lineages have risen 30% in the past two weeks (as of 6 July 2022) and increased in four out of six WHO regions [2]. This warning comes at a time when many countries have still not reached WHO’s vaccination targets (Bulgaria 29.9%, Slovakia 50.8%, Croatia 55.6%, Slovenia 57.1%, Poland 59.6%) [3] or gained access to antivirals. This is also the case for Romania, with a cumulative uptake of full vaccination in the total population of 42.3% as of 14 July 2022 [3]. Moreover, even for those vaccinated, vaccine efficiency declined with the appearance of new mutations in the structure of spike protein, thus amplifying the existent hesitancy in adherence to preventive measures [4]. Breakthrough infections in vaccinated populations with the recently emerged SARS-CoV-2 Omicron variant in the fifth wave of the pandemic exposed some of these limitations of COVID-19 vaccines and highlighted the need for other medical treatments such as drug therapy [5]. However, antiviral drugs prove to reduce the risk for severe outcomes and death in non-hospitalized patients or when given in the first 5 days after COVID diagnosis [6,7,8], but fewer data are on efficiency in hospitalized patients with severe disease [8]. These findings reinforce the need not to underestimate general preventive measures and the role of massive testing.
Worldwide, countries with high vaccination rates validated COVID-19 knowledge and behavior questionnaires (France [9], Spain [10], Italy [11], Hungary [12], Brazil [13], Korea [14], India [15], etc.). Timely insights from behavioral and attitudinal data proved to be crucial for the decision-making process in the COVID-19 international public health policies [16,17].
Currently, there is no validated Romanian questionnaire to assess people’s attitudes toward anti-COVID vaccination and general preventive measures. Such a tool is vital for improving people’s attitudes and compliance and reaching vaccination targets.
In this context, we considered the opportune warning from WHO, and we aligned ourselves to this call by proposing the first validated Romanian questionnaire to assess people’s level of education and attitudes towards general measures to protect against COVID-19 infection. Another objective of our study was to characterize the level of adherence and the main motivations behind acceptance or refusal to follow the imposed preventive measures. The ultimate goal was to use the proposed questionnaire within national programs to identify populational areas with low adherence and establish specific educational programs to increase the willingness to vaccinate against COVID-19 in an effort to prevent a full-blown sixth wave.

2. Materials and Methods

2.1. Questionnaire Development

We designed a survey tool, in the Romanian language, based on the guidelines provided by the World Health Organization Regional Office for Europe (WHO) [18] for the European member states that wish to conduct behavioral insights studies related to COVID-19.
The items of the questionnaire were proposed and reviewed by three physicians. The variables measured by this questionnaire addressed protective measures against COVID-19 infection: social distancing, masking, testing, vaccination, green certificate and following medical advice. As defined by the European Commission, the green certificate is “a proof that a person has been vaccinated against COVID-19, has received a negative test result or has recovered from COVID-19” [19].
The Likert scale was used to assess the answer to each question: 0—“Strongly Disagree”, 1—“Disagree”, 2—“Neither Agree nor Disagree”, 3—“Agree”, 4—“Strongly Agree”. The final score of the test represents the sum of all item scores. The lower the total score, the more people would tend to have a positive attitude toward vaccination and governmental measures.

2.2. Questionnaire Validation

The tested hypothesis was the following: patients with a better questionnaire score on the knowledge of preventive measures will have more frequent correct proangiogenic behavior.
After its development, the validation process followed several steps.

2.2.1. Face Validity

Two experts evaluated the proposed items for their clarity, adequacy and significance in relation to appropriate measuring of knowledge and attitudes toward SARS-CoV-2 infection. Each item was evaluated by experts in relation to three standards:
-
Relevance to the purpose of the questionnaire;
-
Good matching with the recommendations provided to patients by national education programs;
-
The relevance of the measures and behaviors we seek to assess.
Each of these three aspects, corresponding to an item, was assessed on a Likert scale from 1 to 5. Items considered by both experts to be unimportant were deleted.
The questionnaire was then tested for ease of use or ambiguity in a pilot test with ten nursing students. Items with an ambiguous definition of terms detected by the pilot test were removed or rephrased.

2.2.2. Study Design and Participants

An observational study was then conducted on a convenience sample of 194 people. The criteria for inclusion was the completion of a Google form-based questionnaire distributed to the general population through social media (Facebook) between 7 March 2022 and 9 June 2022. All incomplete questionnaires were excluded. Respondents were not paid for their participation. The dissemination of the questionnaire drawn up on Google sheets was conducted through Facebook by the authors.
Approval from the Research Ethics Committee of the “Gr. T. Popa” University of Medicine and Pharmacy Iaşi (Nr. 155/23.02.2022) was obtained before contacting participants and collecting data. All participants provided signed informed consent. The informed consent was included as an introduction to the questionnaire. All data were anonymized to maintain participant confidentiality.

2.2.3. Data Collection

Demographic data (age, gender, educational level, profession, medical history, vaccination history) and questionnaire answers revealing knowledge, attitudes and behaviors related to the COVID-19 outbreak were recorded for each responder.

2.2.4. Psychometric Evaluation

We used psychometric criteria on the data obtained in the study to evaluate the following characteristics:
(a) Items to which at least 5% of people responded in the same way were retained [20,21];
(b) The discrimination index (DI) is a measure of how well an item can differentiate between good candidates and less able ones. For a specific item, DI, expressed as a percentage, was determined by calculating the difference between the number of people in the upper quartile and, respectively, in the lower quartile of the test’s total score, considering only responders who answered correctly to that item. Items with a coefficient of less than 20% do not have a sufficient capacity to discriminate between subjects and were therefore removed [21];
(c) The ability of each element to discriminate between people with different levels of knowledge was measured by correlating the score for each element with the overall test score (TSC—total score correlation). All items with significant correlations at a p < 0.001 were retained [21].

2.2.5. Identifying Underlying Components

The next step was to identify latent components/factors through exploratory factorial analysis. Factorial analysis validates the construction of the questionnaire and reduces the number of variables to a number of components called factors that encompass a subset of the variables sharing a certain communality expressed through multicollinearity.
The Pearson correlation coefficient was determined for conducting the discriminative analysis. In order to evaluate construct validity, we first assessed the homogeneity of variance with the Levene test. Afterward, we determined whether there were significant differences in the mean scores using the ANOVA test. The construct validity was tested using exploratory analysis. The Kaiser–Meyer–Olkin measure of sampling adequacy (KMO) was used to signal in advance whether the sample size is large enough to extract factors reliably. The Bartlett’s Sfericity Test was used to assess the presence of multicollinearity, and the p-value of Bartlett’s test of sphericity was considered significant if it was lower than 0.05. The factor loading value of individual items is set to be greater than 0.25. We considered variables with a correlation coefficient greater than 0.4 as variables that contribute to the formation of a factor.
Principal component analysis (PCA) was used to reduce the dimensionality of the dataset in case of the presence of interrelated variables while retaining as much as possible of the variation.
Several indices were used to assess a good model fit for the construct, including the ratio of chi-square to degree of freedom (χ2/df) < 5.0, root mean square error of approximation (RMSEA) ≤ 0.08, comparative fit index (CFI) > 0.9, Tucker Lewis Index (TLI) > 0.9 and p > 0.05 for the chi-square test [22].

2.2.6. Internal Fidelity

The Cronbach’s alpha coefficient was used to assess internal fidelity. The Guttman Split-Half coefficient showed the probability of test-retest reproducibility.

2.3. Statistical Analysis

The database containing the questionnaire responses was managed with Microsoft Office Excel 2007. Statistical processing was performed using the JASP 0.16.1 [15] and JAMOVI [14] software.
In order to test the normality of the distribution of the questionnaire scores, the Kolmogorov–Smirnov test was applied.
Sample size calculation was undertaken using the single proportion formula: n = Zα/22·P·(1 − p)/d2 at 95% confidence interval, where Zα/2 = 1.96, P = prevalence of 50% and d = 5% of marginal error [23]. We also defined a reference group to estimate the sample size. Due to the lack of other available data, we used a population over 15 years old, with residency in the North East Region of Romania, reported on 1 July 2019 (n = 2,620,857) to define the size of the reference group [17]. Based on these calculations, the minimum sample size required was 370 (95% CI) or 165 (80% CI).
In order to emphasize the conformity and utility of our survey in screening for people’s attitudes toward vaccination, we performed both simple and forwarded binomial logistic regression to predict the vaccination status using the test’s score or other variables resulting from the factor analysis.
For ANOVA analyses, in the presence of two independent groups with α = 0.05 and a power of 95%, the sample size needed is estimated to be 210 participants. In the case of 190 participants, the power decreases to 93%. For logistical regression with 194 participants, OR = 1.3 and critical z 0.99, the power was estimated with the use of the G*Power 3.1.9.7 software at a value of 0.68.

3. Results

Our sample included 194 people, of which 137 (70.6%) were women, with a mean age of 41.5 years old and a standard deviation (SD) of 12.4. Age varied between 18 and 89 years old. Of all respondents, 58.2% inhabited urban areas. Regarding the geographical area, the participants came from the north-eastern part of Romania, and almost half of them were from Iasi county: Iasi 110, Bacau 14, Botosani 24, Galati 13, Neamt 12, Suceava 14, Vaslui 7. A significant proportion of the study sample declared to work in healthcare (27.53%) (Table 1).
The first version of the questionnaire reviewed by experts consisted of 40 items, to which the participants were instructed to respond according to their conviction (Table 2). The score for the items M2, M5, M6, M7, M8, T2, T4, V1, V2, V4, V5, V6, V7, V8, S1, S3, S4, S5, S6, S7, S8, CV2, CV3 and CV4 was reversed.
The mean score of the questionnaire was 42.8 (95% IC: 41–44.6; SD = 12.6). Significant differences in the score were observed between subgroups regarding the area of residence, the presence of diabetes, the awareness of the potential severity of SARS-CoV2 disease and the profession (Table 3). Vaccinated people had significantly lower scores.
According to the psychometric criteria, items SD1, M2, M3, M4, M6, V1, V2, V3, V4, V6, V10, S3, S4, S6 and S7 did not meet the recommendation that at least 5% of people should respond in the same way and were eliminated. Items M2, M3, M4, M6, V2, V3, V6, S3, S4, S6 and S7 were withdrawn due to the DI that was lower than 20%. Item V3 was removed due to the lack of ability to discriminate between people with different levels of knowledge.
The exploratory factorial analysis for the identification of latent components identified three factors that together explained 52.7% of the total variation and were, therefore, chosen as the components of our questionnaire (Table 4, Figure 1). T The value of the Kaiser–Meyer–Olkin coefficient (0.791) and the result of Bartlett’s test (χ2 = 1571.865, p < 0.001) indicated the adequacy of the sample size chosen for our analysis. The names of the factors are given according to the variables that they encompass. Factor 1 was named “Compliance with protective measures”, factor 2 was “Attitudes toward vaccination” and factor 3 was “Attitudes regarding potential COVID-19 therapies”. For factor 1, there are high correlations between items exploring social distancing, testing and the protective role of mask-wearing. For factor 2, the items selected were addressing potential harmful side effects or inefficiency of the vaccines and attitudes toward restrictions associated with the green certificate. For factor 3, the questions investigated behaviors toward potential therapies for SARS-CoV-2 infection, uncertainty toward their scientific validation and distrust in the scientific plausibility of protective measures.
Principal component analysis (PCA) simplified the complexity of high-dimensional data while retaining trends and patterns. After performing PCA and excluding the variables with loadings lower than 0.5, the final model contains 16 items (Table 5 and Table 6) and confirms the factorial structure of the proposed questionnaire with the three dimensions identified by the exploratory factorial analysis.
The value of the alpha Cronbach coefficient shows a good internal fidelity of the questionnaire (0.853). The Guttman Split-Half coefficient shows the probability of test-retest reproducibility. By applying the Split-Half method to show the probability of test-retest reproducibility, a value of the Guttman Split-Half coefficient of 0.859 was obtained, which indicates that the fidelity of the scale is acceptable (Table 7).
As another argument for the survey’s ability to screen people’s attitudes toward vaccination, we identified that the test’s final score predicts the presence of vaccination for a cut-off value of 50% with an accuracy of 0.773 and an AUC of 0.793 (Figure 2).
Table 8 illustrates the predictive power of various variables used in the factorial analysis to estimate the presence of vaccination. In total, the questionnaire score, the diagnosis of diabetes, the advice provided by healthcare workers and the medical profession proved to be significant predictors of vaccination.

4. Discussion

Our paper represents the first validation study of a Romanian questionnaire aimed at screening attitudes toward vaccination and preventive measures against SARS-CoV-2 infection. Our article contributes to the literature with a scientific method to validate a questionnaire in a non-English language. As such, our method can serve as a model to be used by researchers in other countries to develop and validate surveys in their language. Moreover, our article characterizes the level of adherence and the main motivations behind acceptance or refusal to follow the COVID-19 preventive measures of a Romanian population.
Even though we are now in a period of relaxation of the public health measures due to an apparent withdrawal of the epidemic in the first half of the year, recent publications warn about several countries entering the sixth wave of the epidemic [24,25,26,27]. According to the WHO’s latest briefing (6 July 2022), preventive measures and vaccination programs should again be implemented [2].
Several reviews dealing with the worldwide attitudes and hesitancy towards COVID-19 vaccination were published [28,29,30]. In countries with high levels of vaccination, knowledge and level of education proved to have positive associations with attitude and adherence to precautionary measures [23]. On the contrary, in Eastern Europe, a lower overall proportion of vaccine acceptance has been reported [30]. Increasing the vaccination rate is crucial in combating the COVID-19 pandemic, but it requires the prior identification of the underlying causes/specific determinants of hesitancy towards vaccines specific to this region.
Similar to other middle-income countries, Romania faces several barriers and difficulties in re-establishing control measures against the COVID-19 epidemic. The means of controlling the SARS-CoV-2 virus infection spreading were accompanied by economic costs such as unemployment, corporate bankruptcies and a disproportionate impact on less-skilled and less-educated workers. Social costs included increased domestic violence and damaged educational systems [31]. Furthermore, the media attention and uncertainty about how to protect against infection generate contradictory attitudes and behaviors among the population, starting from accepting protective measures and vaccination to denying their effects and even the disease. Along with the conspiracy theories, the imposition of compulsory vaccination has increased social tensions in many countries, with vaccination being associated with the threat to individual human freedoms, such as the right to free choice of health or the right to work. This perception of the threat to freedom has provided an opportunity for various political factions to condemn vaccination, to question the effectiveness of current vaccines amid the tension generated by general protection measures seen as restrictive in order to gain electoral capital. These perspectives emphasize that while COVID-19 vaccines are becoming increasingly available, they are still met with reluctance, and thus, safety measures (e.g., face masks, personal hygiene and social distancing) are still of key importance in protecting personal and public health against COVID-19 [32].
In order to meet the WHO’s recommendations and overcome the particular Romanian patterns of public health measures adherence and acceptance, we developed and validated a questionnaire that addresses knowledge, attitudes and practices focusing on preventive measures against COVID-19. It is divided into six categories related to social distancing, mask-wearing, testing, vaccination, use of supplements and the benefit of a green certificate.
The analysis of the completed questionnaire database offers a global picture of the adherence motivations to general preventive measures against COVID-19 in Romania. By knowing the actual reasons behind not reaching the country’s vaccination target, specific measures can be taken to increase people’s confidence and compliance. For instance, cardiovascular patients within our 194-sized sample did not have higher vaccination rates, although cardiovascular diseases are known risk factors for developing severe forms of COVID-19 [33,34,35,36]. However, the presence of diabetes was significantly associated with a better score in our questionnaire and with a higher proportion of vaccinated people. Thus, specific measures directed toward cardiovascular patients to raise awareness through educational programs could be effective in levering up vaccination rates. Other main determinants of vaccine acceptance and preventive measures compliance, as identified by our study, were related to vaccines’ safety and efficiency, the people’s trust in the government and medical system, and health literacy. These results show us that there is room for progress in COVID-19 health literacy through intensified education. Gaining people’s trust in the government and healthcare system through communication and education should also be a priority. Our results also suggest a need for targeted community awareness interventions for the most vulnerable populations, those with no school education, the elderly and people living in rural areas.
Moreover, we conducted a multivariate analysis that identified several predictors of vaccination adherence: questionnaire’s final score, medical profession, medical advice versus mass media consultation, the presence of diabetes mellitus and the fear of the disease.
In addition to proposing a Romanian survey regarding COVID-19 attitudes, we also proceeded to the scientific validation of the questionnaire in order to obtain pertinent information in a reliable and valid way [21] and ensure that the questionnaire “measures what is intended to be measured” [37]. The application of improper measurement tools that are not validated can lead to inaccurate and misleading findings, resulting in a poor plan for interventions and, therefore, too unreliable efficacy.
Validity is assessed through two categories of tests, which evaluate theoretical construct and empirical construct. The theoretical construct is tested through face validity. The purpose of face validity is to ascertain that the items of the questionnaire fully represent the domain that is intended to be judged [37]. We used literature reviews, critical incidents, direct observations and expert judgment approaches to construct a questionnaire of 40 items with acceptable face validity. The literature review did not identify, to date, any similar validated questionnaire in the Romanian language [38]. Face validity is an important aspect related to the empirical construct of the questionnaire [20,21]. The next validation step was represented by the psychometric evaluation, which prompted the elimination of 15 questionnaire items due to not meeting psychometric criteria. The remaining 25 items were included in the exploratory factor analysis. The factor analysis revealed a three factors structure with 20 items. The resulting model explained 52.7% of the variance in our study. Furthermore, PCA was applied, resulting in a final model of 16 items.
In order to strengthen the evidence that the questionnaire “measures what is intended to be measured”, we proved that the test is able to discriminate between vaccinated and unvaccinated people.
Our results are similar to other resembling COVID-19 questionnaire validations in other languages. A Korean study was based on similar sample size (229) and obtained an alpha Cronbach coefficient of 0.75 for validating a survey that assesses COVID-19 knowledge, attitudes and practices among nursing students [39]. Likewise, the validation of an Indian questionnaire to assess knowledge, attitude, practices and concerns regarding COVID-19 vaccination among the general population was based on 201 participants and resulted in an alpha Cronbach coefficient of 0.86. With the exception of a french questionnaire aimed to assess COVID-19 knowledge and behaviors that was validated on a big cohort of 8045 participants [9], our study fits the current international scientific landscape for the population sample as well as validity results.

Limitations

Firstly, the number of survey respondents was small. This low interest may be explained by the fact that the incidence of COVID-19 cases declined during the spring of 2022, preventive measures were relaxed, and the mass media and people’s attention was focused on the war in Ukraine, with which Romania shares a significant long border. However, this number is still appropriate for factor analysis and testing of the questionnaire. Secondly, our sample might not be representative of the general population in Romania as the proportion of vaccinated people was 74,8%, much higher than that reported by the authorities of 42.3% [3] (as of 14 July 2022). Moreover, a significant proportion were healthcare workers, thus explaining the high rate of vaccination. The percentage of women (70.6%) in our sample is also higher compared to the general population. In our future validation studies, we aim to overcome these limitations by including a larger population with similar variables’ distributions as the general population.

5. Conclusions

In the context of the re-emergence of new variants of SARS-CoV-2 spreading globally, with steady recommendations from the WHO to re-establish some general preventive measures and reinforce vaccination programs, this study aimed to achieve the following: (1) propose the first validated Romanian questionnaire aimed at screening attitudes toward COVID-19 measures; (2) identify the main motivations to accept or reject anti-COVID-19 measures in a Romanian population sample; and (3) pave the way to the integration of our questionnaire as part of the national programs to take the pulse of people’s attitudes, raise awareness and identify populational areas that need specific interventions, in order to reach vaccination targets and prevent a full-blown sixth wave of the COVID-19 pandemic in Romania.

Author Contributions

Conceptualization, A.D.P., S.A.A., A.I.E. and I.V.P.; methodology, A.D.P.; software, A.D.P.; validation, A.B., I.V.P. and R.A.D.; data curation, A.D.P., A.I.E. and R.A.D.; writing—original draft preparation, A.D.P. and I.V.P.; writing—review and editing, A.B., S.A.A., A.D.P. and I.V.P.; visualization, A.I.E. and R.A.D.; supervision, A.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the “Gr. T. Popa”, the University of Medicine and Pharmacy Iaşi (155/23.02.2022).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data used in the study will be available from the corresponding authors upon request.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Wang, X.; Pasco, R.F.; Du, Z.; Petty, M.; Fox, S.J.; Galvani, A.P.; Pignone, M.; Johnston, S.C.; Meyers, L.A. Impact of Social Distancing Measures on Coronavirus Disease Healthcare Demand, Central Texas, USA. Emerg. Infect. Dis. 2020, 26, 2361–2369. [Google Scholar] [CrossRef] [PubMed]
  2. Global COVID Cases Increased by 30% in Past 2 Weeks, 6k Monkeypox Cases Reported: WHO|FULL. Available online: https://www.youtube.com/watch?v=crNF9y4Ixhw (accessed on 14 July 2022).
  3. European Centre for Disease Prevention and Control-COVID-19 Vaccine Tracker. Available online: https://vaccinetracker.ecdc.europa.eu/public/extensions/COVID-19/vaccine-tracker.html#uptake-tab (accessed on 14 July 2022).
  4. Malik, J.A.; Ahmed, S.; Mir, A.; Shinde, M.; Bender, O.; Alshammari, F.; Ansari, M.; Anwar, S. The SARS-CoV-2 mutations versus vaccine effectiveness: New opportunities to new challenges. J. Infect. Public Health 2022, 15, 228–240. [Google Scholar] [CrossRef] [PubMed]
  5. Ng, T.I.; Correia, I.; Seagal, J.; DeGoey, D.A.; Schrimpf, M.R.; Hardee, D.J.; Noey, E.L.; Kati, W.M. Antiviral Drug Discovery for the Treatment of COVID-19 Infections. Viruses 2022, 14, 961. [Google Scholar] [CrossRef] [PubMed]
  6. Kaore, S.; Atal, S. Molnupiravir in COVID-19: A scoping review. Curr. Drug Res. Rev. 2022. Epub ahead of print. [Google Scholar] [CrossRef] [PubMed]
  7. Fatima, M.; Azeem, S.; Saeed, J.; Shahid, A.; Cheema, H.A.; Group, C.-T.S. Efficacy and safety of molnupiravir for COVID-19 patients. Eur. J. Intern. Med. 2022, 102, 118–121. [Google Scholar] [CrossRef]
  8. Najjar-Debbiny, R.; Gronich, N.; Weber, G.; Khoury, J.; Amar, M.; Stein, N.; Goldstein, L.H.; Saliba, W. Effectiveness of Paxlovid in Reducing Severe COVID-19 and Mortality in High Risk Patients. Clin. Infect. Dis. 2022. [Google Scholar] [CrossRef]
  9. Verot, E.; Bonjean, P.; Chaux, R.; Gagnaire, J.; Gagneux-Brunon, A.; Pozzetto, B.; Berthelot, P.; Botelho-Nevers, E.; Chauvin, F. Development and Validation of the COVID-19 Knowledges and Behavior Questionnaire in a French Population (CoVQuest-CC). Int. J. Environ. Res. Public Health 2022, 19, 2569. [Google Scholar] [CrossRef]
  10. Paredes, B.; Cárdaba, M.; Cuesta, U.; Martinez, L. Validity of the Spanish Version of the Vaccination Attitudes Examination Scale. Vaccines 2021, 9, 1237. [Google Scholar] [CrossRef]
  11. Bruno, F.; Laganà, V.; Pistininzi, R.; Tarantino, F.; Martin, L.; Servidio, R. Validation and psychometric properties of the Italian Vaccination Attitudes Examination (VAX-I) scale. Curr. Psychol. 2022, 27, 1–11. [Google Scholar] [CrossRef]
  12. Kotta, I.; Kalcza-Janosi, K.; Szabo, K.; Marschalko, E.E. Development and Validation of the Multidimensional COVID-19 Vaccine Hesitancy Scale. Hum. Vaccines Immunother. 2022, 18, 1–10. [Google Scholar] [CrossRef]
  13. Da Cunha de Sá-Caputo, D.; Sonza, A.; Bachur, J.A.; Bernardo-Filho, M. Development, validation and reliabilty of a questionnaire to evaluate the changes on the level of physical exercises and in daily life habits due to COVID-19 pandemic social distancing. Acta Bio-Med. Atenei Parm. 2020, 91, e2020004. [Google Scholar] [CrossRef]
  14. Seong, M.; Park, J.; Chung, S.; Sok, S. Development of the Adult Pandemic Attitude Scale (A-PAS). Int. J. Environ. Res. Public Health 2021, 18, 6311. [Google Scholar] [CrossRef] [PubMed]
  15. Agarwal, A.; Ranjan, P.; Rohilla, P.; Saikaustubh, Y.; Sahu, A.; Dwivedi, S.N.; Aakansha; Baitha, U.; Kumar, A. Development and validation of a questionnaire to assess preventive practices against COVID-19 pandemic in the general population. Prev. Med. Rep. 2021, 22, 101339. [Google Scholar] [CrossRef] [PubMed]
  16. Perrotta, D.; Grow, A.; Rampazzo, F.; Cimentada, J.; Del Fava, E.; Gil-Clavel, S.; Zagheni, E. Behaviours and attitudes in response to the COVID-19 pandemic: Insights from a cross-national Facebook survey. EPJ Data Sci. 2021, 10, 17. [Google Scholar] [CrossRef] [PubMed]
  17. Gallè, F.; Sabella, E.A.; Roma, P.; Ferracuti, S.; Da Molin, G.; Diella, G.; Montagna, M.T.; Orsi, G.B.; Liguori, G.; Napoli, C. Knowledge and Lifestyle Behaviors Related to COVID-19 Pandemic in People over 65 Years Old from Southern Italy. Int. J. Environ. Res. Public Health 2021, 18, 10872. [Google Scholar] [CrossRef]
  18. Monitoring Knowledge, Risk Perceptions, Preventive Behaviours and Trust to Inform Pandemic Outbreak Response. Available online: https://www.euro.who.int/__data/assets/pdf_file/0007/436705/COVID-19-survey-tool-and-guidance.pdf (accessed on 14 July 2022).
  19. European Commission-Questions and Answers -Digital Green Certificate. Available online: https://ec.europa.eu/commission/presscorner/detail/en/qanda_21_1187 (accessed on 1 September 2022).
  20. Parmenter, K.; Wardle, J. Evaluation and Design of Nutrition Knowledge Measures. J. Nutr. Educ. 2000, 32, 269–277. [Google Scholar] [CrossRef]
  21. Taherdoost, H. Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/Survey in a Research. SSRN Electron. J. 2016, 5, 28–36. [Google Scholar] [CrossRef]
  22. Goni, M.D.; Naing, N.N.; Hasan, H.; Wan-Arfah, N.; Deris, Z.Z.; Arifin, W.N.; Baaba, A.A.; Njaka, S. A confirmatory factor analysis of the knowledge, attitude and practice questionnaire towards prevention of respiratory tract infections during Hajj and Umrah. BMC Public Health 2020, 20, 1684. [Google Scholar] [CrossRef]
  23. Devkota, H.R.; Sijali, T.R.; Bogati, R.; Clarke, A.; Adhikary, P.; Karkee, R. How Does Public Knowledge, Attitudes, and Behaviors Correlate in Relation to COVID-19? A Community-Based Cross-Sectional Study in Nepal. Front. Public Health 2020, 8, 589372. [Google Scholar] [CrossRef]
  24. COVID Czar: Israel in Midst of Pandemic’s Sixth Wave, but No Restrictions Pending. Available online: https://www.timesofisrael.com/covid-czar-israel-in-midst-of-pandemics-sixth-wave-but-no-restrictions-pending (accessed on 14 July 2022).
  25. Wave 6-COSMO Canada-Implementing the WHO Behavioural Insights Tool on COVID-19 to Inform Response Efforts, Including Policy, Interventions and Communications. Available online: https://impact.canada.ca/en/cosmo-canada/wave-6 (accessed on 14 July 2022).
  26. Alberta Past 6th COVID-19 Wave, Health Minister Says. Available online: https://www.cbc.ca/news/canada/edmonton/alberta-covid-omicron-ba2-monkeypox-1.6483636 (accessed on 14 July 2022).
  27. Top NY Doctor: New COVID Wave Is Starting, With the ‘Worst Version’ of Omicron. Available online: https://www.nbcnewyork.com/news/coronavirus/top-ny-doctor-new-covid-wave-is-starting-with-the-worst-version-of-omicron/3752858 (accessed on 14 July 2022).
  28. Cascini, F.; Pantovic, A.; Al-Ajlouni, Y.; Failla, G.; Ricciardi, W. Attitudes, acceptance and hesitancy among the general population worldwide to receive the COVID-19 vaccines and their contributing factors: A systematic review. eClinicalMedicine 2021, 40, 101113. [Google Scholar] [CrossRef]
  29. Sallam, M. COVID-19 Vaccine Hesitancy Worldwide: A Concise Systematic Review of Vaccine Acceptance Rates. Vaccines 2021, 9, 160. [Google Scholar] [CrossRef] [PubMed]
  30. Popa, A.D.; Enache, A.I.; Popa, I.V.; Antoniu, S.A.; Dragomir, R.A.; Burlacu, A. Determinants of the Hesitancy toward COVID-19 Vaccination in Eastern European Countries and the Relationship with Health and Vaccine Literacy: A Literature Review. Vaccines 2022, 10, 672. [Google Scholar] [CrossRef]
  31. Su, Z.; Wen, J.; McDonnell, D.; Goh, E.; Li, X.; Segalo, S.; Ahmad, J.; Cheshmehzangi, A.; Xiang, Y.T. Vaccines are not yet a silver bullet: The imperative of continued communication about the importance of COVID-19 safety measures. Brain Behav. Immun. Health 2021, 12, 100204. [Google Scholar] [CrossRef] [PubMed]
  32. Lahiri, A.; Jha, S.S.; Chakraborty, A.; Dobe, M.; Dey, A. Role of Threat and Coping Appraisal in Protection Motivation for Adoption of Preventive Behavior During COVID-19 Pandemic. Front. Public Health 2021, 9, 678566. [Google Scholar] [CrossRef]
  33. Kufel-Grabowska, J.; Bartoszkiewicz, M.; Ramlau, R.; Litwiniuk, M. Cancer patients and internal medicine patients attitude towards COVID-19 vaccination in Poland. Adv. Clin. Exp. Med. 2021, 30, 805–811. [Google Scholar] [CrossRef] [PubMed]
  34. Gheorghe, A.S.; Negru, S.M.; Nitipir, C.; Mazilu, L.; Marinca, M.; Gafton, B.; Ciuleanu, T.E.; Schenker, M.; Dragomir, R.D.; Gheorghe, A.D.; et al. Knowledge, attitudes and practices related to the COVID-19 outbreak among Romanian adults with cancer: A cross-sectional national survey. ESMO Open 2021, 6, 100027. [Google Scholar] [CrossRef] [PubMed]
  35. Matovina Brko, G.; Popovic, M.; Jovic, M.; Radic, J.; Bodlovic Kladar, M.; Nikolic, I.; Vidovic, V.; Kolarov Bjelobrk, I.; Kukic, B.; Salma, S.; et al. COVID-19 vaccines and cancer patients: Acceptance, attitudes and safety. J. BUON 2021, 26, 2183–2190. [Google Scholar]
  36. Puteikis, K.; Mameniskiene, R. Factors Associated with COVID-19 Vaccine Hesitancy among People with Epilepsy in Lithuania. Int. J. Environ. Res. Public Health 2021, 18, 4374. [Google Scholar] [CrossRef]
  37. Bolarinwa, O.A. Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger. Postgrad. Med. J. 2015, 22, 195–201. [Google Scholar] [CrossRef]
  38. Tsang, S.; Royse, C.F.; Terkawi, A.S. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J. Anaesth. 2017, 11, S80–S89. [Google Scholar] [CrossRef]
  39. Park, D.-I. Development and Validation of a Knowledge, Attitudes and Practices Questionnaire on COVID-19 (KAP COVID-19). Int. J. Environ. Res. Public Health 2021, 18, 7493. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Scree plot of the eigenvalues of the factors identified in our questionnaire.
Figure 1. Scree plot of the eigenvalues of the factors identified in our questionnaire.
Healthcare 10 01827 g001
Figure 2. The performance of the test’s final score to predict the status of vaccination.
Figure 2. The performance of the test’s final score to predict the status of vaccination.
Healthcare 10 01827 g002
Table 1. Characteristics of the study sample.
Table 1. Characteristics of the study sample.
Socio-Demographical Characteristic Count% of Total N% of Total
GenderF13770.6 M5729.4
Area of residenceRural8141.8Urban11358.2
ProfessionHealthcare5927.353Other13572.7
Formal educationSecondary/high-school10754.87Bachelor or more8845.12
Cardiovascular diseaseYes4221.6No15278.4
Pulmonary diseasesYes136.7No18193.3
Diabetes mellitusYes6131.4No13368.6
SmokingYes6131.4No13368.6
AllergyYes3518No15982
VaccinatedYes14474.2No5025.8
Type of vaccineBNT162b211982.6
mRNA-127332.08
AZD1222 (ChAdOx1)96.25
NJ-78436735 (Ad26.COV2.S)139.02
History of COVID-19Yes9850.5No9649.5
Hospitalised Yes1616.32No8283.67
SupplementsYes11845.36No7639.17
Vitamin D1210.16
Zinc32.54
Vitamin C1714.40
Multivitamins8672.88
COVID-19 treatmentUmifenovir55.1No3232.65
Favipiravir22
Ivermectin11.02
Recommended by the doctor5859.18
Source of adviceFacebook105.2
Family115.7
Internet7438.1
Family doctor2713.9
TV3417.5
Friends42.1
Other physician3417.5
COVID-19 can put me at risk?Yes10453.6No9046.4
When I think about the possibility of getting COVID-19, I feel worried.Yes7840.2No11659.8
Table 2. First version of the questionnaire.
Table 2. First version of the questionnaire.
Answer Variants: Number of Respondents (Percentage)
CodeItem12345TSCDI
1SD1Adherence to social distancing can prevent SARS-CoV-2 infection94 (48.5)40 (20.6)41 (21.1)6 (3.1)13 (6.7)0.60820.61
2SD2The recommended distance to avoid the spread of the virus is 1.5 m76 (39,2)53 (27,3)38 (19.6)13 (6.7)14 (7.2)0.45427.31
3SD3I believe that I have followed this recommendation most of the time58 (29.9)58 (29.9)38 (21.6)13 (9.3)14 (7.2)0.45829.89
4SD4I avoided crowded areas64 (33)45 (23.2)44 (22.7)23 (11.9)16 (9.3)0.39923.19
5M1Wearing a mask prevents SARS-CoV-2 infection83 (42.8)43 (22.2)38 (19.6)11 (5.7)19 (9.8)0.58622.16
6M2The mask is correctly worn by covering only the mouth34 (17.5)7 (3.6)8 (4.1)14 (7.2)131 (67.5)0.46111.34
7M3The mask is correctly worn by covering both the nose and the mouth156 (80.4)33 (17)22 (11.3)8 (4.1)30 (15.5)0.5360
8M4I wear the mask correctly in public/crowd most of the time101 (52.1)33 (17)22 (11.3)8 (4.1)30 15.5)0.52211.34
9M5I can’t stand the mask because I feel suffocated/can’t speak or hear well33 (17)19 (9.8)23 (12.9)25 (12.9)94 (48.5)0.48824.74
10M6I can’t wear the mask because of comorbidities29 (14.9)8 (4.1)10 (5.2)9 (4.6)138 (71.1)0.5189.79
11M7The mask prevents me from recognizing the people around me41 (21.1)28 (14.4)36 (18.6)25 (12.9)64 (23)0.55131.4
12M8Children should not wear a mask41 (21.1)23 (11.9)40 (20.8)26 (13.4)63 (32.3)0.54434.02
13T1I agree with PCR testing91 (46.9)41 (21.1)30 (15,5)10 (5.2)22 (11.3)0.54421.1
14T2Nasal tests are invasive36 (18.6)24 (12.4)47 (24.2)25 (12.9)62 (32)0.30237.11
15T3The RT-PCR test is based on the detection of viral RNA76 (39.2)38 (19.6)50 (25.8)11 (5.7)19 (9.8)0.40545.3
16T4Many RT-PCR tests are false35 (18)19 (9.8)38 (19.6)44 (22.7)58 (29.9)0.55742.26
17T5I agree with rapid tests73 (37.6)42 (21.6)44 (22.7)13 (6.7)22 (11.3)0.49821.64
18T6Rapid tests can give erroneous results73 (37.6)49 (25.3)49 (25.3)14 (7.2)9 (4.6)0.25325.25
19V1I didn’t get vaccinated because it affects my freedom32 (16.5)6 (3.1)16 (8.2)13 (6.7)127 (65.5)0.60014.94
20V2I didn’t get vaccinated because it changes my DNA, and it’s an experiment on the population32 (16.5)7 (3.6)18 (9.3)12 (6.2)125 (64.4)0.61415.46
21V3I got vaccinated for my protection and to protect those around me109 (56.2)18 (9.3)26 (13.4)6 (3.1)35 (19)0.0929.27
22V4I did not get vaccinated because the vaccine is not tested enough, and I am waiting to see what happens to others43 (22.2)7 (3.5)20 (10.3)16 (8.2)108 (55.7)0.39913.9
23V5I got vaccinated so as not to have problems at work or to be able to travel abroad43 (22.2)24 (12.)41 (21.1)12 (6.2)74 (38.8)0.48827.31
24V6I did not get vaccinated because I have chronic diseases or allergies.18 (9.3)5 (2.6) 20 (10.3)9 (4.6)142 (73.20.4814.63
25V7Vaccinated people transmit the disease more or similar to unvaccinated people41 (21.1)14 (7.2)58 (29.9)26 (13.4)54 (27.8)0.47158,13
26V8The vaccine can cause heart disease (myocarditis/endocarditis)/thrombosis)36 (18.6)27 (13.9)53 (27.3)29 (14.9)49 (25.3)0.52353.14
27V9The vaccine does not provide protection against infection49 (25.3)30 (15.5)41 (21.1)30 (15.5)44 (22.7)0.29636.59
28V10The vaccine prevents severe or fatal COVID-19135 (69.8)54 (27.8)03 (1.5)2 (1)0.52381
29S1Foods rich in vitamin C surely prevent COVID-1920 (10.3)20 (10.3)64 (33)46 (23.7)44 (22.7)0.29632.98
30S2Vitamin D has an important role in disease prevention27 (13.9)34 (17.5)76 (39.3)28 (14.4)29 (14.9)0.17239.17
31S3Quercetin is a drug that cures COVID-1913 (6.7)11 (5.7)97 (50)24 (12.4)49 (25.3)0.45412.37
32S4Zinc cures COVID-1918 (9.3)10 (5.2)68 (35.1)30 (15.5)68 (35.1)0.52815.46
33S5The state does not provide drugs that cure COVID-1933 (17)27 (13.9)62 (32)38 (19.6)34 (17.5)0.48131.95
34S6Arbidol is effective in curing COVID-1928 (14.4)21(10.8)98 (50.5)22 (11.3)25 (12.9)0.39710.82
35S7Although Arbidol is very good, it is not brought to hospitals24 (12.4)22 (11.3)106 (54.6)19 (9.8)23 (11.9)0.5050
36S8Ivermectin is effective in curing COVID-1921 (10.8)13 (6.7)108 (55.7)17 (8.8)35 (18)0.38222.68
37CV1The green certificate is a measure to control the spread of the infection56 (28.9)34 (17.5)35 (18)17 (8.8)52 (26.8)0.54844.32
38CV2I don’t think using the green certificate prevents the infection from spreading62 (32)29 (14.9)30 (15.5)27 (13.9)44 (22.7)0.58059.13
39CV3The green certificate is a measure against the freedom of individuals51 (26.3)21 (10.8)35 (18)19 (9.8)68 (35.1)0.62735.1
40CV4The green certificate should not be used at this time66 (34)19 (9.8)35(18)22 (11.3)52 (26.4)0.61435.09
TSC—total score correlation; DI—discrimination index.
Table 3. Differences in score between subgroups.
Table 3. Differences in score between subgroups.
Socio-Demografic Breakdown of Study SampleNMeanSDCohen’s dp
GenderFeminine13742.5712.720.060.682
Masculine5743.4213.32
Area of residenceRural8145.0012.650.290.046
Urban11341.2612.86
Cardiovascular diseasesYes 4243.5911.040.070.628
No15242.6113.36
DiabetesYes 3946.3011.890.340.049
No10541.9413.00
COVID infection is a threat for meYes 10439.5911.250.55<0.001
No9046.5513.66
VaccinateYes 14439.3511.261.15<0.001
No5052.8212.06
History of COVID infectionYes 9843.0211.970.020.832
No9642.6513.79
Medical ProfessionYes13538.6511.231.31<0.001
No5953.3711.26
Source of adviceFamily and friends1538.6010.02 0.08
Mass media11744.3612.49
Medical professionals6240.9813.87
Table 4. The 3 identified components/factors of the questionnaire along with the variables that they encompass.
Table 4. The 3 identified components/factors of the questionnaire along with the variables that they encompass.
Item Factor 1Factor 2Factor 3
SD40.777
SD30.744
SD20.650
T10.630
M10.612
T50.538
V8 0.800
V7 0.659
CV2 0.601
CV3 0.566
V9 −0.535
CV4 0.504
CV1 0.453
S1 0.690
S2 −0.619
M8 0.493
S5 0.439
S8 0.417
M7 0.416
T2, T3, T4, T6 and V5 have factor loadings lower than 0.4.
Table 5. Goodness of fit indices for the final model.
Table 5. Goodness of fit indices for the final model.
FactorsNo of ItemsModel Fit for the Construct Goodness of Fit Indices
χ2 (df)p ValuedfCFITLIRMSEASRMR
Model 118397.0 (164)<0.0011640.8110.7810.0860.086
Model 215176.76 (112)<0.0011120.9380.9250.0550.056
CFI—comparative fit index; TLI—Tucker Lewis Index; RMSEA—root mean square error of approximation; SRMR—standardized root mean squared error.
Table 6. Final model’s factor loadings.
Table 6. Final model’s factor loadings.
FactorIndicatorSymbolEstimateStd. Errorz-Valuep95% Confidence IntervalStd. Est. (All)
LowerUpper
1SD3λ110.7250.0917.974<0.0010.5470.9030.577
SD4λ120.7260.0957.632<0.0010.5390.9120.557
T1λ130.9530.09310.206<0.0010.7701.1360.704
T5λ140.8650.0949.179<0.0010.6801.0490.647
M1λ150.9660.08910.908<0.0010.7931.1400.741
SD2λ160.7240.0878.275<0.0010.5520.8950.594
2V8λ210.6120.1065.776<0.0010.4040.8200.434
V9λ22−0.7040.111−6.370<0.001−0.921−0.487−0.472
CV1λ231.0130.1109.220<0.0010.7981.2290.644
CV2λ241.1340.10810.503<0.0010.9221.3450.713
CV3λ251.1120.1129.941<0.0010.8931.3310.684
CV4λ261.1880.10910.931<0.0010.9751.4010.736
V7λ270.6370.1115.761<0.0010.4200.8540.432
3M7λ311.0150.1228.328<0.0010.7761.2540.666
M8λ321.0200.1228.367<0.0010.7811.2590.669
S5λ330.6250.1065.894<0.0010.4170.8330.478
Table 7. Frequentist scale reliability statistics.
Table 7. Frequentist scale reliability statistics.
EstimateCronbach’s αGuttman’s λ2Guttman’s λ6MeanSD
Point estimate0.8530.8590.88542.82512.877
95% CI lower bound0.8200.8220.86541.01311.710
95% CI upper bound0.8810.8870.91544.63714.303
Table 8. Role of variables in predicting vaccination status.
Table 8. Role of variables in predicting vaccination status.
PredictorEstimateSEZpOdds Ratio95% Confidence Interval
LowerUpper
Intercept−6.63601.2136−5.468<0.0010.00131<0.0010.0142
Sum Score0.08070.02133.793<0.0011.084031.0401.1302
Area of residence
Urban–Rural−0.12730.4311−0.2950.7680.880500.3782.0498
Diagnosis of diabetes
Yes–No1.23280.4924 2.5030.0123.430921.3079.0071
COVID-19 can endanger me?
Yes–No0.29080.46510.6250.5321.337490.5383.3280
Where did you find out the most about COVID infection?
1–2 (family–mass-media)0.12920.87630.1470.8831.137900.2046.3393
3–2 (medical workers–media)1.78840.52393.414<0.0015.980122.14216.6975
Profession (medical–nonmedical)
2–11.96580.48504.053<0.0017.140742.76018.4760
SE = standard error of the coefficient estimates.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Popa, A.D.; Antoniu, S.A.; Enache, A.I.; Popa, I.V.; Dragomir, R.A.; Burlacu, A. Development and Validation of a Questionnaire to Assess Knowledge and Attitudes toward COVID-19 Preventive Measures in Romania. Healthcare 2022, 10, 1827. https://doi.org/10.3390/healthcare10101827

AMA Style

Popa AD, Antoniu SA, Enache AI, Popa IV, Dragomir RA, Burlacu A. Development and Validation of a Questionnaire to Assess Knowledge and Attitudes toward COVID-19 Preventive Measures in Romania. Healthcare. 2022; 10(10):1827. https://doi.org/10.3390/healthcare10101827

Chicago/Turabian Style

Popa, Alina Delia, Sabina Antonela Antoniu, Armand Iustinian Enache, Iolanda Valentina Popa, Raluca Alina Dragomir, and Alexandru Burlacu. 2022. "Development and Validation of a Questionnaire to Assess Knowledge and Attitudes toward COVID-19 Preventive Measures in Romania" Healthcare 10, no. 10: 1827. https://doi.org/10.3390/healthcare10101827

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop