Performance change with the number of training data: A case study on the binary classification of COVID-19 chest X-ray by using convolutional neural networks

https://doi.org/10.1016/j.compbiomed.2022.105251Get rights and content

Highlights

  • A binary classification for COVID-19 and Normal was conducted by using CNNs.

  • Two available large datasets consisting of 8000 chest X-ray images were used.

  • AUC, sensitivity, and specificity were evaluated with the increasing training data.

  • Performances were rapidly improved and reached an equilibrium state.

  • The fine-tuning method improved all performances.

Abstract

One of the features of artificial intelligence/machine learning-based medical devices resides in their ability to learn from real-world data. However, obtaining a large number of training data in the early phase is difficult, and the device performance may change after their first introduction into the market. To introduce the safety and effectiveness of these devices into the market in a timely manner, an appropriate post-market performance change plan must be established at the timing of the premarket approval. In this work, we evaluate the performance change with the variation of the number of training data. Two publicly available datasets are used: one consisting of 4000 images for COVID-19 and another comprising 4000 images for Normal. The dataset was split into 7000 images for training and validation, also 1000 images for test. Furthermore, the training and validation data were selected as different 16 datasets. Two different convolutional neural networks, namely AlexNet and ResNet34, with and without a fine-tuning method were used to classify two image types. The area under the curve, sensitivity, and specificity were evaluated for each dataset. Our result shows that all performances were rapidly improved as the number of training data was increased and reached an equilibrium state. AlexNet outperformed ResNet34 when the number of images was small. The difference tended to decrease as the number of training data increased, and the fine-tuning method improved all performances. In conclusion, the appropriate model and method should be selected considering the intended performance and available number of data.

Keywords

Performance change
Deep learning
Convolutional neural network
COVID-19
Chest X-ray
Medical device regulation

Cited by (0)

View Abstract