PLOS ONE conducted a post-publication assessment of this article [1] due to questions about aspects of the peer review process. In the post-publication assessment, which involved a member of PLOS ONE’s Editorial Board and an external reviewer, concerns were raised about the rigor and appropriateness of the statistical analyses and whether the article’s conclusions are well-supported by the data.

This research was conducted in the first year of the global COVID pandemic. This does not alter the journal’s requirements with regard to research reporting standards and publication criteria, but in this context and given the objectives of the study the PLOS ONE Editors understand that there were constraints which may reasonably have impacted study design decisions. The authors commented that when the article was written, “many insights [had] yet to be developed and data were scarce. This fundamentally influenced our conceptual framework and the chosen methodological approach.”

Even so, based on the outcome of the post-publication assessment, PLOS ONE has concerns about the robustness of the results and the strength of evidence supporting the article’s conclusions due to the study design limitations. Specific concerns include: (a) limited variables were used to represent readiness structures, responses, and outcomes, (b) the data analysis approach did not include a robust statistical analysis, and (c) potential biases in the dataset and study design were not adequately addressed or discussed in the article [1].

Overall, the outcome of the assessment was that the results reported in the original published article did not adequately support the conclusions. The PLOS ONE Editors issue this Expression of Concern to notify readers of these issues and additional information and analyses that were provided in post-publication discussions.

As is discussed below and in S1 File, the authors provided additional context, clarifications, and analyses in post-publication discussions that partially resolved the concerns and lend additional support for the article’s main claims. With this provided, the journal considers that this body of work makes a valid contribution to the published literature, although not all concerns have been fully resolved.

Primary data underlying the study’s results and other supplementary materials (referred to below as “OSF Supp. Materials”) are available at https://osf.io/g69hy/?view_only=9c54713bd17c4dac8c8c1a6104d67d45.

Below, we outline the main concerns raised in this case and the authors’ response to each. For additional information, see S1 File.

1. The conceptual framework is limited in its scope of what determines society readiness and reactions. The article did not adequately address the rationale for aspects of the study design, or provide evidence to support that the included factors provide a reasonable, reliable representation of the context. For example, concerns were raised about the scientific validity and appropriateness of representing health risk factors by a singular variable (the percentage of population aged 65 or more), and why health risk of age was used as an indicator of society readiness for a public health emergency.
   With regard to health risk factors, the authors commented that when the study was conducted–during early stages of the pandemic–unambiguous research and data on risk factors were limited, the authors viewed age as the only reliable indicator known at that time. The article’s conclusions (final paragraph of [1] discuss this limitation and note that the choice of dimensions should be revisited in the future; for example, future studies could combine characteristics of population (age, gender, obesity) and medical conditions (prevalence of diabetes, cardiovascular diseases, etc.).
   The authors also explained that use of age as a health risk indicator in the study was in line with the conceptual framework of the study: societal readiness seen as demographic predisposition for a more severe outcome of the pandemics and societal reaction in terms of occurrence of government interventions targeting elderly population.
2. The statistical analyses presented in the article are not suitable to address the aims of this study, or for use in analyzing outcomes that represent a cumulative distribution function. The analyses did not adequately address key confounds, such as under detection of COVID cases, different case definitions, and the timing of epidemic waves and first reports of COVID cases within the country being analyzed. Also, the study did not include sensitivity analyses, and Tables 3 and 5 did not present analysis of variance across the countries in a transparent way. Due to the lack of statistical rigor applied in these data analyses, the claims reported in the article based on the correlation analysis are not adequately supported.
   The authors explained that their approach to this study was not rooted in statistical modeling or an approach where independent variables are evaluated to see how they affect outcomes. Instead, the study was designed based on an approach similar to theory-based evaluation which strives to systematically link concept (dimensions within proposed conceptual framework) and data (indicators derived in operationalization phase). This conceptual analytical model relies on so called ’empirical indicants’; a small subset of possible indicators used to represent a particular concept [2]. The authors further noted that the results are not dependent on or sensitive to the order to dimension inclusions, and that the correlation analysis (Table 4) should be understood only as a “showcase performed to initially check possible redundancy between variables” i.e. Table 4 is not intended to demonstrate relationships between variables based on statistical modeling analyses.
   The authors also provided the following clarifications about the analysis method used in the study:
   • ELECTRE MLO was used as an analytical support because it has been shown to be effective for evaluation of multidimensional phenomena and as an alternative to common composite indexes.
   • This approach is suitable for mix-data type and has meaningful interpretation of performance differences, and ELECTRE based analysis have been previously applied in epidemic and pandemic research [3–6]).
   • The method was used to demonstrate the usability of proposed framework, and to obtain relative positioning of countries according to the depicted dimensions of readiness and reaction. Only basic elements of ELECTRE-MLO were used due to constraints of the study design (e.g. variables used, data quality, time frame). The outcome reflected in three color-coded groups of countries was used to monitor movements in hierarchy obtained by outranking method and was not intended to offer a “forecasting tool in terms of the initial severity of the COVID-19”.
3. The article did not adequately address biases in the study design and/or analyses, discuss what steps were taken to address, minimize, or otherwise mitigate effects of bias, or to discuss impacts of potential biases as limitations of the study. For example, the article does not address how global or regional responses might have affected country-level responses, preparedness, or outcomes; the data collected for each country may have been biased due to the relative timings within the study period of within-country COVID exposures and waves of positivity and COVID-related mortality; and issues pertaining to under-detection and differences in case definition need to be addressed in analyses to examine the reliability and robustness of the results.
   The authors commented that per the reported analysis countries’ responses were not uniform despite the global context and information available about responses in other countries. Some governments started with early measures well before their first case. For example, Croatia, Bulgaria, and Norway introduced measures (GovI1) as early as the end of January, whereas others did not take action until their first cases were reported. In an effort to account for differences in country-level exposure timings, the authors defined three sub-indicators related to government measures: timeliness, strictness, and duration of government interventions. The authors noted that although COVID deaths are a lagging indicator, since all countries from the sample were at the peak of infection at a similar time (March-April), death counts declined during May (OSF Supp. materials/Examples/picture 4.).
   Country-level COVID timelines are in OSF Supp. materials/Timeline.
4. Apple mobility data do not provide a consistent representation of population-level mobility across countries due to differences in iPhone usage. The authors noted that data about iPhone users in the sample range from the lowest in Poland (5% of all mobile phone users i.e. approx. 1.5–2 million users) and Serbia (10% of all mobile phone users i.e. approx. 400.000–500.000 users) to the highest Sweden (50% of all mobile phone users) and UK (50% of all mobile phone users) [OSF Supp. Materials/Apple mobility trends]. They conducted a comparison of Google and Apple tracking data for Poland and Serbia (countries with the lowest portion of iPhone users in the sample) and found a correlation of ~94% (OSF Supp. materials/Examples/figure 6.)). These data should be interpreted with caution since Apple and Google use different methodologies to track movement, but this finding supports that users of iPhones vs. other devices may have comparable mobility patterns.
5. The study’s main conclusions reported in the Abstract and the ‘Discussion and conclusions’ section of [1] overstate what can be drawn from this study’s data. The penultimate sentence of the Abstract (“Our findings confirm…”) is to be disregarded. The ‘Discussion and conclusions’ section of the article is hereby updated with the following new text, which replaces paragraph 2 (with all three bullet points) and paragraphs 6 to 10 (from “If we recall our correlation analysis…” to the paragraph ending with reference 44).