Introduction

A new infectious disease detected in Wuhan, Hubei province, China, at the end of 2019, spread throughout the whole world and rapidly caused a “pandemic.” The World Health Organization reported SARS-CoV-2, a virus of the Coronaviridae family, to be the infection agent [1, 2]. This agent leads to severe pneumonia, called COVID-19, in humans.

During this period, anxiety and maladaptive behaviors increased in patients undergoing chronic immunosuppressive therapy with the thought that there was a lack of immune response against viruses [3]. Patients with rheumatic diseases may feel greater concern as the characteristics of the disease can make them more vulnerable to infectious diseases. In addition, the fact that patients are actively using non-steroidal anti-inflammatories, corticosteroids, and synthetic and biological disease-modifying agents may increase these concerns [4]. Therefore, it is inevitable that patients will increasingly seek alternative sources of health-related information.

In the digitalized world, web-based platforms are probably the most easily accessible health-related information sources for the community. YouTube has grown into one of the most popular social platforms in the world with more than one billion Internet consumers and 6 billion hours of viewing time every month [5]. YouTube includes a large number of videos on the pathogenesis, prevention, diagnosis, and treatment options of several health problems [6,7,8,9]. YouTube videos related to Ebola were viewed millions of times during the 2014 outbreak [10]. During disease outbreaks, online platforms should be considered a “double edged sword,” which can lead to positive and negative consequences. While the dissemination of accurate YouTube-based information may contribute to controlling the outbreak, misleading information may cause panic in society and increase the spread of disease. YouTube’s main handicap in this regard is that it does not include a regulatory mechanism to control the content quality of the uploaded videos.

To the best of our knowledge, there have been no previous studies assessing YouTube videos providing information on COVID-19 related to rheumatic diseases. Therefore, the aim of this study was to evaluate the quality of YouTube videos about COVID-19 in relation to rheumatic diseases. It was also aimed to identify video sources providing high-quality information and to determine which topics were covered more in these videos.

Materials and methods

This was planned as a descriptive study, and the possible search terms were discussed in the first stage. A total of six search terms were determined as “COVID-19 rheumatology, SARS CoV-2 rheumatology, coronavirus 2019 rheumatology, COVID-19 arthritis, SARS CoV-2 arthritis, and coronavirus 2019 arthritis.” These search terms were used on YouTube (www.youtube.com) on 2 April 2020 to determine videos that could be evaluated in the study. The browser search history was deleted before the study to minimize the effect of past Internet use on search results. Video listings are made based on view counts which enables the most viewed videos to be listed on the first page. Although there are studies assessing all videos, the most common method is to work on a fixed sample size [11]. It has been reported that over 90% of Internet users review videos on the first 3 pages of query results [12]. Therefore, the videos on the first three pages (60 videos) for each search term were reviewed. The 360 videos were recorded in a file for future analysis because search results on YouTube are constantly changing. This sample selection method has been used in similar studies in literature [6, 13, 14]. Videos in a language other than English, duplicate videos, off topic videos, and videos without audio were excluded from the study.

Assessment of quality

The quality of the videos was assessed using the Global Quality Scale (GQS) which is a 5-point scale of 1–5 points and has been used for quality assessments in similar studies [6, 9, 13, 15]. Video flow, usefulness, and quality can be evaluated using the GQS, with scores applied of 4–5 points indicating high quality, 3 points indicating moderate quality, and 1–2 points indicating low quality. The following scoring system was used in this study:

  • 1 point: The video is of poor quality, poor flow, lacking most information, and therefore not useful for patients.

  • 2 points: The video is generally of poor quality, and although some information is given, it is of limited use for patients.

  • 3 points: The video is of moderate quality, and some important information is sufficiently discussed. In these videos, accurate and incorrect information are presented in a balanced manner. However, high-quality information is provided together with misleading information.

  • 4 points: The video is of good quality and good flow. The video is useful for patients, covering the most relevant information and presenting accurate information to a large extent, but it may include minor deficiencies.

  • 5 points: The video is of excellent quality and excellent flow and is very useful for patients. These videos include completely accurate information [16].

All the videos were independently evaluated by two physicians (BFK, MSA). The Kappa coefficient was used for the determination of inter-rater consistency. Videos with inconsistency in terms of GQS scores (between BFK and MSA) were identified. These were then evaluated by another physician (ARS), and consensus was reached.

Assessment of reliability

The modified DISCERN tool (DS) was applied to assess reliability. The modified DS was adapted from the original version and includes five yes-no questions [15]:

  • Is the video clear, concise, and understandable?

  • Are reliable sources of information used?

  • Is the information presented balanced and unbiased?

  • Are additional sources of information listed for patient reference?

  • Are areas of uncertainty/controversy mentioned?

Each “yes” answer is scored as 1 point and each “no” answer as 0 point. Thus, the score ranges from 0 to 5 points with higher scores indicating greater reliability.

Video parameters

The upload date, video duration, view counts, likes, dislikes, and comments were recorded for all videos. The values for views per day were calculated by dividing the total number of views by the total number of days on YouTube. The like ratio was calculated as like / [like + dislike].

Video sources

Video sources were categorized under eight headings: (1) society/non-profit organization, (2) physician, (3) health-related website, (4) university/academic, (5) patient/independent user, (6) non-physician healthcare personnel, (7) pharmaceutical company, and (8) news agency.

Coding of video content

It was recorded whether the video addressed the following topics: symptoms of COVID-19, relationship between rheumatological diseases and risk of COVID-19, whether to continue the use of existing rheumatological drugs, COVID-19 prevention practices for those with rheumatological disease, the place of hydroxychloroquine in the treatment of COVID-19, conditions that should be consulted to a doctor or hospital, and psychological effects of COVID-19 on rheumatology patients.

Ethics statement

Videos which are publicly available on YouTube were assessed, and no human participants/animals were included. Therefore, Ethics Committee approval was not required for this study. Similar studies have also followed the same path [6, 17, 18].

Statistical analysis

Data obtained in the study were analyzed statistically using the Statistical Package for the Social Sciences version 20.0 software (SPSS Inc., Chicago, IL, USA). Descriptive data were stated as number, percentage, and median (minimum-maximum) values. Conformity of the data to normal distribution was checked using the Shapiro-Wilk test. The agreement between the two physicians was assessed using the Kappa coefficient. Comparisons between low-, moderate-, and high-quality groups were performed using the Kruskal-Wallis test. A value of p < 0.05 was accepted as statistically significant.

Results

Of the total 360 videos, 176 off topic videos, 113 repetitive videos, 22 videos in a language other than English, and 3 videos with sound quality problems were excluded from the study. After application of the inclusion and exclusion criteria, a total of 46 videos were identified for evaluation in the study. The median duration of the videos was 403.5 min (range, 31–3699 min). The median view count and number of comments were 589.5 (range, 21–635,561) and 2.5 (range, 0–1972), respectively. The general features of the videos are presented in Table 1.

Table 1 General features of the videos
Full size table

Of the videos, 41.4% (n = 19) were of high quality, 21.7% (n = 10) were of moderate quality, and 36.9% (n = 17) were of low quality according to the GQS. The Kappa score for inter-rater agreement was 0.74. Of the videos, 100% (n = 2) produced by academics/universities and 56.3% (n = 9) produced by physicians were of high quality. Low-quality videos were determined as 100% (n = 4) of those produced by patients/independent users and 87.5% (n = 7) of the videos produced by news agencies. The distribution of the video quality groups according to the uploading sources is presented in Table 2.

Table 2 Categorization of the videos according to sources, n (%)
Full size table

Significant differences were detected between the quality groups in terms of DS scores and views per day (p < 0.0001, p = 0.004). The highest scores were found in the high-quality group. No significant differences were determined in respect of comments per day (p = 0.139) and like ratios (p = 0.232) according to video quality (Table 3).

Table 3 DS, views per day, comments per day, and like ratio of videos according to video quality
Full size table

When the content of the videos was examined, 45.6% (n = 21) provided information on “the place of hydroxychloroquine in the treatment of COVID-19,” and 41.3% (n = 19) provided information on “whether to continue the use of existing rheumatological drugs.” The distribution of the videos according to content is presented in Table 4.

Table 4 Distribution of the video contents
Full size table

Discussion

YouTube is a powerful and free platform which can reach a large part of society during a pandemic. As a result of rapid adaptation to digital life, the use of Internet platforms such as YouTube can be expected to increase as a source of medical information. However, the structure of YouTube allows it to present videos providing high-quality/useful information and misleading/inaccurate information at the same time. The spread of misleading/inaccurate information during a pandemic can increase anxiety and panic, leading to undesirable results. Therefore, it was considered that it would be useful to evaluate Internet-based information during this COVID-19 pandemic. The study results demonstrate that less than half of the YouTube videos focusing on the relationship between COVID-19 and rheumatological diseases were of high quality. The main sources of the high-quality videos were academics/universities and physicians, whereas the videos presented by patients/independent users and news agencies were of predominantly low quality.

Of the total videos, 41.4% (n = 19) were of high quality, 21.7% (n = 10) were of moderate quality, and 36.9% (n = 17) were of low quality. Khatri et al. [19] evaluated YouTube videos on the novel coronavirus outbreak and reported that 67% of the videos in English were useful. During the Zika virus pandemic and the H1N1 influenza pandemic, 70.3% and 61.3% of the videos, respectively, were reported to be useful and informative [20, 21]. However, there are also studies reporting lower rates of high-quality/useful videos outside of the pandemic [13, 22]. In the current study, several factors may have played a role in detecting relatively lower rates of high-quality/useful videos compared with other pandemic studies. In other studies, the pandemic was evaluated as a whole, whereas in this study, there was particular focus on the relationship between the pandemic and rheumatic diseases. This may have affected the results. The current study sample was relatively small compared with other pandemic studies, and the assessment of videos is subjective, and different criteria may have been used in other evaluations.

The main sources of the high-quality videos were academics/universities, followed by physicians. Low-quality videos were predominantly sourced by patients/independent users and news agencies. In a study of the Zika virus pandemic, Bora et al. [20] reported that the main sources of informative videos were universities and misleading videos were produced by independent users. In contrast to the current study results, news agencies were found to be a source of informative videos. Khatri et al. [19] declared news agencies as the main source of useful videos and individual users as the source of misleading videos. Şahin et al. [23] stated that videos produced by independent users were of lower quality than videos from healthcare professionals. This study reveals the importance of source when reviewing videos to obtain medical information from YouTube. Although there is inconsistency in respect of news agencies, academics/universities and healthcare professionals should be considered the source of high-quality information. In the current study, all the videos from academics/universities were of high quality, but the number of videos was very low. Therefore, universities, academics, and healthcare professionals should be supported and motivated to produce videos that provide accurate, useful, and unbiased medical information.

In addition to video quality, one of the major concerns is that videos providing misleading information may have a greater number of views. Although the results of the current study do not support this concern, low-quality YouTube videos during the Zika virus and H1N1 influenza pandemics were found to attract more Internet users than high-quality videos [20, 21]. YouTube studies conducted on Sjögren’s syndrome and rheumatoid arthritis have demonstrated that Internet users tended to view useful videos [15, 24]. Users diagnosed with chronic rheumatic disease such as rheumatoid arthritis and Sjögren’s syndrome may be better able to distinguish misleading information compared with general users during an acute pandemic, as they already have a basic knowledge of rheumatological diseases. Additionally, high reputation of academics, universities, and physicians may have led the patients to keep the videos presented by these sources in the foreground. Therefore, high-quality videos may be more viewed.

The most frequently discussed topics in videos were the place of hydroxychloroquine in the treatment of COVID-19 and whether to continue the use of existing rheumatological drugs. Interest in hydroxychloroquine has increased following studies presenting potential positive effects in the treatment of COVID-19 [25, 26]. Another reason for the high frequency is that this attractive topic was presented in almost all of the videos provided by news agencies. The first question in the mind of rheumatology patients after the outbreak of COVID-19 has been the condition of current drug treatments. Considering the potential effects of rheumatology drugs on the immune system, it can be understood why this issue has been frequently discussed in the videos.

YouTube studies have some limitations. Although the videos were not evaluated by a single researcher, the scoring is subjective. The videos in a single snapshot were listed, and the evaluations were made of this video list. YouTube is a dynamic platform, and over time new videos are added, commented on, and viewed. The video search was performed only in English. As a video search can be influenced by the search history, this issue was minimized by deleting the browser search history. Geographic location and language potentially affect the search results. Finally, the relatively small sample size can be considered another limitation.

Conclusion

Although the ratio of high-quality videos was high, a considerable number of low-quality videos were detected. At this point, the importance of video sources becomes evident in order to select high-quality videos and access correct information. Academics, universities, and healthcare professionals should create videos on YouTube to ensure that Internet users can access more useful, high-quality, and accurate information.