https://orcid.org/0000-0002-3993-7920
Figures
Abstract
Objective
To evaluate the reporting quality of randomized controlled trials (RCTs) regarding patients with COVID-19 and analyse the influence factors.
Methods
PubMed, Embase, Web of Science and the Cochrane Library databases were searched to collect RCTs regarding patients with COVID-19. The retrieval time was from the inception to December 1, 2020. The CONSORT 2010 statement was used to evaluate the overall reporting quality of these RCTs.
Results
53 RCTs were included. The study showed that the average reporting rate for 37 items in CONSORT checklist was 53.85% with mean overall adherence score of 13.02±3.546 (ranged: 7 to 22). The multivariate linear regression analysis showed the overall adherence score to the CONSORT guideline was associated with journal impact factor (P = 0.006), and endorsement of CONSORT statement (P = 0.014).
Conclusion
Although many RCTs of COVID-19 have been published in different journals, the overall reporting quality of these articles was suboptimal, it can not provide valid evidence for clinical decision-making and systematic reviews. Therefore, more journals should endorse the CONSORT statement, authors should strictly follow the relevant provisions of the CONSORT guideline when reporting articles. Future RCTs should particularly focus on improvement of detailed reporting in allocation concealment, blinding and estimation of sample size.
Citation: Yin Y, Shi F, Zhang Y, Zhang X, Ye J, Zhang J (2021) Evaluation of reporting quality of randomized controlled trials in patients with COVID-19 using the CONSORT statement. PLoS ONE 16(9): e0257093. https://doi.org/10.1371/journal.pone.0257093
Editor: Daoud Al-Badriyeh, Qatar University, QATAR
Received: April 24, 2021; Accepted: August 23, 2021; Published: September 23, 2021
Copyright: © 2021 Yin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The study was funded by Horizontal project of School of Public Health of Lanzhou University “Research on the Thinking, difficulties and Countermeasures of High-quality Development of Lanzhou Hand and Foot Surgery Hospital” (071100278), Gansu Science and Technology Plan (Innovation Base and Talent Plan) project (21JR7RA607), and Health industry scientific research project of Gansu Province (GSWSKY-2019-50), received by Fugui Shi and Juxia Zhang.
Competing interests: The authors have declared that no competing interests exist.
Introduction
With the development of evidence-based medicine, randomized controlled trials (RCTs) became the gold standard to compare the effectiveness of different interventions [1]. It can avoid possible bias in clinical trial design, balance confounding factors and improve the effectiveness of statistical tests [2]. Therefore, a complete and accurate report will enable readers to fully assess the authenticity of results [2]. If the RCT report is unsatisfactory, the validity of trials will be reduced [3], which may adversely affect the results of meta-analyses and recommendations for clinical practice [4].
To improve the reporting quality of RCTs, the CONSORT (reporting standards of randomized controlled trials) statement was developed in 1996 [5], revised in 2001 [6] and 2010 [7]. The updated CONSORT statement includes 25 entries that provide specific guidance for RCT reports. Since many of the 25 items were subdivided into two subitems, the list actually consists of 37 items [7]. It is currently known to be endorsed by over 600 biomedical journals and endorsed by several prominent editorial organizations including the International Committee of Medical Journal Editors (ICMJE) and the World Association of Medical Editors (WAME) [8]. After update of CONSORT statement, the overall reporting quality of RCTs has improved, but there were still many RCT reports with various deficiencies [9, 10]. A study by Yao et al. of 65 RCTs related to ophthalmic surgery showed the mean CONSORT score was 8.9 (range 3.0–14.7) and the reporting quality was quite low [11]. A study of 71 RCTs regarding herbal interventions showed that the compliance rates of CONSORT checklist in these RCTs ranged from 0% to 97.18% [12]. Poor reporting quality of RCTs is the major barrier to evidence-based practices, as it can distort the available evidences in the medical literature, and prevent clinical decision-makers from obtaining true results from trials [13]. Thus, it is critical for researchers to build well-reported standard of RCTs. On the one hand, it can ensure the validity of clinical trials and the authenticity and scientificity of research results [14]. On the other hand, it is conducive to conducting secondary studies, such as meta-analyses and systematic reviews [15].
With the outbreak of the COVID-19 in 2019, in order to quickly and effectively control the epidemic, a large number of RCTs regarding COVID-19 have been published. However, the reporting quality of these RCTs is unclear, to our knowledge, no study has specifically evaluated the reporting quality of RCTs regarding patients with COVID-19.
The primary objective of our study was to assess the reporting of RCTs regarding patients with COVID-19 and analyze possible related factors, so as to provide theoretical basis for subsequent studies and meta-analyses.
Methods
Ethical review
Ethical approval was not necessary for this study, as the study did not involve patients and included RCTs can be traced from databases.
Search strategy
PubMed, Embase, Web of Science and the Cochrane Library databases were searched to collect RCTs regarding patients with COVID-19. The retrieval time was from the inception to December 1, 2020. The search was conducted by two investigators and the detailed strategy was shown in S1 File.
Study selection
Studies meeting the following criteria were enrolled in the study: (1) Randomized controlled trial. (2) The confirmed or suspected patients of COVID-19 according to the diagnostic criteria of "the latest Clinical guidelines for novel coronavirus" issued by the World Health Organization (WHO). (3) Interventions related to patients or suspected patients.
Exclusion criteria included: (1) Animal experiments, reviews, systematic reviews, case reports. (2) Repeated publications. (3) The abstract or full text is not available.
The titles of the retrieved article were imported into the Endnote X9 and screened by 2 reviewers independently. We first reviewed the title and abstract of each article and decided to regard its appropriateness for inclusion. In case of doubt, we downloaded full texts to judge whether an article was RCT. Any disagreement was solved by consensus.
Data extraction
Two authors independently extracted the general characteristics and reporting data of included studies into Excel, any discrepancy was resolved through discussion. The general characteristics include continent of first-author, number of authors, sample size, age and type of participants, the type of interventions, journal impact factor and journals’ endorsement of the CONSORT statement.
Assessment of reporting quality
The CONSORT statement was chosen as a tool to assess reporting quality of these RCTs [7]. We assessed the compliance of each RCT by 25 items of CONSORT statement, each checklist item and subitem was answered with “yes” or “no”. According to the above items, the coincidence rate of each item of 53 studies was counted one by one. A point for the item being granted if all sub-sections were answered as yes, if one of two subsections was reported, a score of 0.5 was awarded, then total score of each study was calculated [1]. Items 3b, 6b, and 14b are not necessarily applicable, if relevant, the article will be graded according to the above guidelines, if the article did not apply to this item, no points was deducted [3].
Statistical analysis
Descriptive statistics was performed to describe general characteristics of 53 studies and the reporting rate of each checklist item/subitem. We used the k coefficient to determine the degree of agreement between reviewers. T-test and ANOVA were used for univariate analysis, multiple linear regression analyses were used to determine the association between potential predictors and reporting quality. All significant predictors in the univariable analyses were entered individually into a multivariable analysis. No significant violation of normality was found in assessments of the residuals. Chi-square tests and Fisher’s exact tests were used to analyze the relationship between journals impact factor and endorsement of CONSORT guideline, T-test was used to analyze the differences between different journal submission requirements and CONSORT score. For all analyses, the statistical significance level was set at P<0.05. Statistical analysis was carried out using SPSS version 21.
Results
Search results
Initially, 8700 articles were obtained, excluding duplicates, 6,922 studies was remained. After screening the titles and abstracts, 198 potentially eligible articles were identified. Subsequently full text of each article was retrieved, and 53 RCTs were confirmed for further assessment. Fig 1 outlined the search detail via the PRISMA flow diagram.
Agreement of reviewers
In the pilot study, inter-observer concordance for article selection had a kappa score of 0.78, which was 0.89 after resolving all disputed items by a discussion with the third reviewer (ZJX), suggesting that inter-observer reliability was almost perfect.
Characteristics of included studies
Among the 53 RCTs, the first author of most RCTs were from Asia and accounted for 62.26%, half of the studies had a sample size greater than 100, 50 studies (94.34%) used drug interventions. 22 (42.51%) studies had an impact factor of more than 10. Nearly half of the studies were published in journals that did not explicitly require authors to follow the CONSORT statement. As showed in Table 1.
The evaluation result of the CONSORT checklist
The study showed that the average reporting rate for 37 items in CONSORT checklist was 53.85%, with 31.35% for the methodological section. Items with the reporting rate of more than 80% were abstract, background, eligibility criteria for participants, outcomes, statistical methods, participant flow chart, baseline data, limitations, generalisability and funding. Except for non-essential items, the remaining items with the reporting rate of less than 20% were trial design, sample size, allocation concealment, implementation, blinding, ancillary analyses and protocol. Only 7 studies (13.21%) reported the methods of masking concealment and 9 studies (16.98%) reported the details of the blinding. Table 2 outlined the reporting frequency of each checklist item.
The CONSORT average score for all study was 13.02±3.546, 95%CI (12.06–13.97).
Factors of effecting overall reporting score
Univariate results showed that CONSORT score was associated with sample size (P<0.001), types of intervention (P = 0.017), journal impact factor (P<0.001), and endorsement of CONSORT statement (P<0.001) (Table 1).
Table 3 displayed the results of multiple linear regression analysis. The four predictors were entered into a multivariable model (constant = 7.779, R2 = 0.533, adjusted R2 = 0.494, P<0.001). Among these, journal impact factor (P = 0.006), and journal with endorsement of CONSORT statement (P = 0.014) still persisted as noticeable predictors of reporting quality.
Journal impact factors less than 10 and more than 10 had statistically significant differences in endorsement of the CONSORT statement (P<0.001) (Table 4). There was a statistically significant difference in reporting quality between journals that required submission of CONSORT checklist and those that did not (P = 0.031) (Table 5).
Discussion
To our knowledge, this is the first study to evaluate the reporting quality of RCTs regarding patients with COVID-19 by the CONSORT statement [7], this will provide important information for clinical decision makers. Our study showed that the overall reporting quality of RCTs regarding patients with COVID-19 was suboptimal. It is clear that the reporting quality of these studies needs to be improved, particularly in terms of methodology section.
Although the CONSORT statement was established to ensure the completeness and accuracy of RCT reports, our study found that some authors still reported their data selectively in a biased way, with the average reporting rate of 53.85%, none of the RCTs provided complete information as required, similar results have been found in other studies [11, 16–18]. This could be due to the large number of patients with COVID-19 emerging in a short period of time, in order to present positive results of various treatment regimens to readers as soon as possible, researchers may have paid more attention to the results of study than report specifications.
The key items with lower reporting rates were mainly concentrated in trial design, sample size, allocation concealment, and blinding. The complete description of trial design can provide readers with accurate research ideas and enable readers to better evaluate the trial results [19], but our study showed that only 5 (9.43%) studies reported the type of trial design. The neglect of two most important items in the methodology section (allocation concealment and details of blinding) was particularly worrisome, as these items are important information to ensure the authenticity of results [20]. Only 7 (13.21%) studies described the methods of allocation concealment, 9 (16.98%) studies reported the details of blinding. We found that some authors tend to write “single” or “double” blind rather than specifying exactly who were unaware of treatment identities. The low reporting rates of these items may reflect the lack of relevant knowledge of researchers to some extent, because the report of sequence generation and concealment of allocation need to have certain knowledge of clinical research methodology [21]. There was evidence that trials with inadequate or unclear allocation concealment overestimated the treatment effects up to 7% [22], and a meta-epidemiological study of blinding showed that unblinded RCTs overestimated the outcome effect by 0.56 standard deviations [23]. In spite of this, some studies still reported these items poorly [11, 17, 24]. A previous study showed that only 12% and 21% of RCTs reported the details of allocation concealment and blinding in four high-impact general medical journals [25]. What’s more, a study of acute herpes zoster showed that none of RCTs reported blinding and masking [16]. Therefore, it is urgent for researchers to pay attention to the design and implementation of allocation concealment and blinding, in order to minimize measurement bias and improve the reporting quality of RCTs. The estimation of sample size can avoid false negative results between the intervention and control groups [26]. Our study showed that only 5 studies (9.43%) reported the details of sample size, which was similar to the results in the field of vascular and endovascular surgery, herpes zoster and plastic surgery [9, 16, 27]. Lack of the reporting of sample size estimation can prevent readers to verify the validity of the trial results [27], researchers should attach great importance to the report of sample size estimation, so as to provide scientific evidence for future clinical studies.
Our study showed that reporting quality was associated with journal impact factor and endorsement of the CONSORT statement, similar results have been found in other areas [15, 27–29]. Studies have shown that 80% of RCTs published in journals that do not endorse CONSORT guideline had defective reporting specifications [30]. We found that journals with impact factor of more than 10 have a higher endorsement of the CONSORT guideline. However, even though some journals have endorsed the CONSORT guideline, the reporting quality of published RCTs was still suboptimal, this may be due to a problem with the entry point for paper submission. We found that some journals only encourage researchers to follow the CONSORT guideline, but do not require authors to submit a CONSORT checklist, which was more common in journals with lower impact factor, this may be because journals with lower impact factor have less stringent policies for accepting and publishing papers. In contrast, journals with higher impact factor have stricter requirements for submission of articles, requiring authors to upload CONSORT checklist when submitting RCT, which forces researchers to write RCTs according to standard reporting specifications. Our study showed that articles that uploaded the CONSORT checklist had higher reporting quality than those that did not have strict submission requirements, therefore, strict submission requirements are the premise to improve the reporting quality of RCTs.
Academic journals are the main media for carrying and publishing papers, the relevant provisions in the manuscript contract will directly affect the quality of published papers [28, 30]. Thus, we suggest that editors should carefully assess whether their journals’ submission requirements are normative, journals should not only endorse the CONSORT guideline, but more importantly require authors to upload the CONSORT checklist as key material for the initial screening when submitting their RCTs. Similarly, peer reviewers should check the completeness and accuracy of CONSORT checklist when reviewing RCTs, editorial boards should also increase their oversight of the entire process from submission to publication, articles of lower reporting quality will not be published. However, in addition to problems in the whole process of submission requirements, review and publication, another potential reason for the poor reporting quality of RCTs is that the CONSORT statement was not publicized enough, researchers lack the awareness of report guideline [31]. A survey of the authors of 101 studies found that only 3% acknowledged the importance of RCT reports and followed the CONSORT guideline when writing papers [31], this suggests that improving researchers’ awareness of the CONSORT guideline is critical to improve the reporting quality of RCTs. Therefore, on the one hand, journals should vigorously promote the CONSORT guideline and can add relevant knowledge of RCT reports to their subscription feeds. On the other hand, research institutions should also increase training in these problems to improve the reporting quality of RCTs.
Limitations of this study
There are some limitations in our study. Although the literature retrieval, screening and quality evaluation were carried out simultaneously by two researchers, there was still some subjectivity. In addition, we only included RCTs of patients with COVID-19 from 4 databases, it could not represent the overall reporting quality of RCTs of COVID-19.
Conclusions
The primary objective of our study was to provide readers a broad overview of the reporting characteristics of RCTs regarding patients with COVID-19. The overall reporting quality of these RCTs was suboptimal, thereby diminishing their potential usefulness, and it can not provide valid evidence for clinical decision-making and systematic reviews. Better reporting quality was associated with higher journal impact factor and endorsement of the CONSORT statement. More journals should endorse the CONSORT statement, authors should strictly follow the relevant provisions of the CONSORT guideline when writing the paper. Future RCTs should particularly focus on improvement of detailed reporting in allocation concealment, blinding and estimation of sample size.
Acknowledgments
We thank Dr. Bin Ma (Evidence-Based Medicine Center, Institute of Traditional Chinese and Western Medicine, School of Basic Medical Sciences, Lanzhou University, Lanzhou, Gansu, China) for providing assistance with editing the final manuscript.
References
- 1. Hajibandeh S, Hajibandeh S, Antoniou GA, Green PA, Maden M, Torella F. Reporting and Methodological Quality of Randomised Controlled Trials in Vascular and Endovascular Surgery. European journal of vascular and endovascular surgery: the official journal of the European Society for Vascular Surgery. 2015;50(5):664–70. pmid:26298221
- 2. Butcher NJ, Mew EJ, Monsour A, Chan AW, Moher D, Offringa M. Outcome reporting recommendations for clinical trial protocols and reports: a scoping review. Trials. 2020;21(1):620. pmid:32641085
- 3. Karpouzis F, Bonello R, Pribicevic M, Kalamir A, Brown BT. Quality of reporting of randomised controlled trials in chiropractic using the CONSORT checklist. Chiropractic & manual therapies. 2016;24:19. pmid:27284400
- 4. Ntala C, Birmpili P, Worth A, Anderson NH, Sheikh A. The quality of reporting of randomised controlled trials in asthma: a systematic review. Primary care respiratory journal: journal of the General Practice Airways Group. 2013;22(4):417–24. pmid:24248328
- 5. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. Jama. 1996;276(8):637–9. pmid:8773637
- 6. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Jama. 2001;285(15):1987–91. pmid:11308435
- 7. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ (Clinical research ed). 2010;340:c332. pmid:20332509
- 8.
CONSORT website. http://www.consort-statement/about-consort/endorses. [cited 18 March 2021].
- 9. Hajibandeh S, Hajibandeh S, Antoniou GA, Green PA, Maden M, Torella F. Association between bibliometric parameters, reporting and methodological quality of randomised controlled trials in vascular and endovascular surgery. Vascular. 2017;25(2):196–207. pmid:27252347
- 10. Huang YQ, Traore K, Ibrahim B, Sewitch MJ, Nguyen LHP. Reporting quality of randomized controlled trials in otolaryngology: review of adherence to the CONSORT statement. Journal of otolaryngology—head & neck surgery = Le Journal d’oto-rhino-laryngologie et de chirurgie cervico-faciale. 2018;47(1):34. pmid:29764496
- 11. Yao AC, Khajuria A, Camm CF, Edison E, Agha R. The reporting quality of parallel randomised controlled trials in ophthalmic surgery in 2011: a systematic review. Eye (London, England). 2014;28(11):1341–9. pmid:25214001
- 12. Pratoomsoot C, Sruamsiri R, Dilokthornsakul P, Chaiyakunapruk N. Quality of reporting of randomised controlled trials of herbal interventions in ASEAN Plus Six Countries: a systematic review. PloS one. 2015;10(1):e108681. pmid:25633206
- 13. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8):iii, ix-xi, 1–193. pmid:20181324
- 14. Aslam A, Imanullah S, Asim M, El-Menyar A. Registration of Clinical Trials: Is it Really Needed? N Am J Med Sci. 2013;5(12):713–5. pmid:24404555
- 15. Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013;257(5):818–23. pmid:23407296
- 16. Qian G, Zeng J, Lu L, Pei W, Liu K, Luo Z, et al. Evaluation of Reporting Quality in Randomised Controlled Trials of Acupuncture for Acute Herpes Zoster by the CONSORT Statement and STRICTA Guidelines. Evidence-based complementary and alternative medicine: eCAM. 2020;2020:4308380. pmid:32148542
- 17. Chen J, Li Z, Liu B, Gan X, Li C, Yu H. Quality improvement in randomized controlled trial abstracts in prosthodontics since the publication of CONSORT guideline for abstracts: a systematic review. Journal of dentistry. 2018:23–9. pmid:29742449
- 18. Vassar M, Jellison S, Wendelbo H, Wayant C, Gray H, Bibens M. Using the CONSORT statement to evaluate the completeness of reporting of addiction randomised trials: a cross-sectional review. BMJ open. 2019;9(9):e032024. pmid:31494625
- 19. Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. Jama. 2013;309(8):814–22. pmid:23443445
- 20. Seehra J, Wright NS, Polychronopoulou A, Cobourne MT, Pandis N. Reporting quality of abstracts of randomized controlled trials published in dental specialty journals. J Evid Based Dent Pract. 2013;13(1):1–8. pmid:23481004
- 21. Chen Y, Li J, Ai C, Duan Y, Wang L, Zhang M, et al. Assessment of the quality of reporting in abstracts of randomized controlled trials published in five leading Chinese medical journals. PLoS One. 2010;5(8):e11926. pmid:20689853
- 22. Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health technology assessment (Winchester, England). 2012;16(35):1–82. pmid:22989478
- 23. Hróbjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. International journal of epidemiology. 2014;43(4):1272–83. pmid:24881045
- 24. Agha RA, Camm CF, Doganay E, Edison E, Siddiqui MR, Orgill DP. Randomised controlled trials in plastic surgery: a systematic review of reporting quality. European journal of plastic surgery. 2014;37(2):55–62. pmid:24707112
- 25. Ghimire S, Kyung E, Kang W, Kim E. Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials. 2012;13:77. pmid:22676267
- 26. Weber EJ, Hoo ZH. Why sample size estimates? Emergency medicine journal: EMJ. 2018;35(12):755–6. pmid:30154143
- 27. Stevanovic A, Schmitz S, Rossaint R, Schürholz T, Coburn M. CONSORT item reporting quality in the top ten ranked journals of critical care medicine in 2011: a retrospective analysis. PloS one. 2015;10(5):e0128061. pmid:26020246
- 28. Canagarajah NA, Porter GJ, Mitra K, Chu TSM. Reporting quality of surgical randomised controlled trials in head and neck cancer: a systematic review. European archives of oto-rhino-laryngology: official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS): affiliated with the German Society for Oto-Rhino-Laryngology—Head and Neck Surgery. 2021. pmid:33604748
- 29. Bigna JJ, Noubiap JJ, Asangbeh SL, Um LN, Sime PS, Temfack E, et al. Abstracts reporting of HIV/AIDS randomized controlled trials in general medicine and infectious diseases journals: completeness to date and improvement in the quality since CONSORT extension for abstracts. BMC medical research methodology. 2016;16(1):138. pmid:27737631
- 30. Hopewell S, Altman DG, Moher D, Schulz KF. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal ’Instructions to Authors’. Trials. 2008 Apr 18;9:20. pmid:18423021
- 31. Reveiz L, Villanueva E, Iko C, Simera I. Compliance with clinical trial registration and reporting guidelines by Latin American and Caribbean journals. Cad Saude Publica. 2013 Jun;29(6):1095–100. pmid:23778541