JMIR Publications

ABSTRACT

Background:

Contact tracing has been globally adopted in the fight to control the infection rate of COVID-19. Thanks to digital technologies, such as smartphones and wearable devices, contacts of COVID-19 patients can be easily traced and informed about their potential exposure to the virus. To this aim, several mobile applications have been developed. However, there are ever-growing concerns over the working mechanism and performance of these applications. The literature already provides some interesting exploratory studies on the community’s response to the applications by analyzing information from different sources, such as news and users’ reviews of the applications. However, to the best of our knowledge, there is no existing solution that automatically analyzes users’ reviews and extracts the evoked sentiments. We believe such solutions combined with a user-friendly interface can be used as a rapid surveillance tool to monitor how effective an application is and to make immediate changes without going through an intense participatory design method which, although in normal circumstances is optimal, but not optimal in emergency situations where a mobile device needs to be deployed immediately with little to no user input from the beginning for the greater public good.

Objective:

In this paper, we aim to analyze the efficacy of AI models and Natural Language Processing (NLP) techniques in automatically extracting and classifying the polarity of users’ sentiments by proposing a sentiment analysis framework to automatically analyze users’ reviews on COVID-19 contact tracing mobile applications. We also aim to provide a large-scale annotated benchmark dataset to facilitate future research in the domain. As a proof of concepts, we also develop a potential web application, based on the proposed solutions, with a user-friendly interface to automatically analyze and classify users’ reviews on the COVID-19 contact tracing applications. The proposed framework combined with the interface which is expected to help the community in quickly analyzing users’ perception about such mobile applications and can be used as a rapid surveillance tool to monitor effectiveness of mobile applications and to make immediate changes without going through an intense participatory design method in emergency situations.

Methods:

We propose a pipeline starting from manual annotation via a crowd-sourcing study and concluding on the development and training of AI models for automatic sentiment analysis of users’ reviews. In detail, we collected and annotated a large- scale dataset of Android and iOS mobile applications users’ reviews for COVID-19 contact tracing. After manually analyzing and annotating users’ reviews, we employed both classical (i.e., Naïve Bayes, SVM, Random Forest) and deep learning (i.e., fastText, and different transformers) methods for classification experiments. This resulted in eight different classification models.

Results:

We employed eight different methods on three different tasks achieving up to an average F1-Scores 94.8% indicating the feasibility and applicability of automatic sentiment analysis of users’ reviews on the COVID-19 contact tracing applications. Moreover, the crowd-sourcing activity resulted in a large-scale benchmark dataset composed of 34,534 reviews manually annotated from the contract tracing applications of 46 distinct countries. The resulted dataset is also made publicly available for research usage.

Conclusions:

The existing literature mostly relies on the manual/exploratory analysis of users’ reviews on the application, which is a tedious and time-consuming process. Moreover, in the existing studies, generally, data from fewer applications are analyzed. In this work, we showed that AI and NLP techniques provide good results in analyzing and classifying users’ sentiments’ polarity, and that the automatic sentiment analysis can help in analyzing users’ responses to the application more quickly with a significant accuracy. Moreover, we also provided a large-scale benchmark dataset composed of 34,534 reviews from 47 different applications. We believe the presented analysis, dataset, and the proposed solutions combined with a user-friendly interface can be used as a rapid surveillance tool to analyze and monitor mobile applications deployed in emergency situations leading to rapid changes in the applications without going through an intense participatory design method.