Assessment of SARS-CoV-2 rapid antigen tests

Introduction

Since the beginning of the “corona pandemic” in spring 2020, the National Coronavirus Testing Strategy in Germany has been constantly adapted depending on the situation in order to meet the health, economic and social challenges of the COVID-19 pandemic. In vitro diagnostics providers and users are also involved in these challenges, but may be pursuing different interests. In this context, rapid SARS-CoV-2 antigen tests (PoC-AgTs) have recently gained importance in the management of the pandemic. However, it is important that a distinction be made between the rapid tests that are used by medical professionals in all sectors of care and those that are intended to be used as tests for self-administration by medical laypersons.

The Paul Ehrlich Institute (PEI), with the participation of the Robert Koch Institute (RKI), has defined qualitative requirements with minimum criteria for point-of-care antigen tests (PoC-AgTs) in a dossier that is continuously being updated [1]. The Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte, BfArM) has compiled an online list of all PoC-AgTs that fulfil these criteria [2]. In its epidemiological bulletin, the RKI recently demonstrated the effects of these quality criteria with regard to the clinical significance of positive and negative test results as well as sensitivity, specificity and prediction probabilities. The latter are particularly dependent on the quality of the performance specifications in low-incidence situations [3].

According to the German Coronavirus Test Ordinance (TestV), only tests that appear on this list may be reimbursed. Already within the first weeks, more than one hundred (n=165, as of 17.11.2020) PoC-AgTs were listed. However, the publication of usable or reimbursable PoC-AgTs does not relieve the user from critically examining the usage instructions before using an in vitro diagnostic product on a patient. Instructions for use therefore remain the mandatory basis for every user to assess the suitability of a product before use.

In an article in the journal Deutsches Ärzteblatt of 30 October 2020, the opportunities and risks of speed in the development of rapid antigen tests were already discussed [4]. The following basic evaluation illustrates which performance data test providers currently provide to users of PoC-AgTs. Currently (as of 02.03.2021), there are more than two hundred PoC-AgTs on the BfArM list, which does not significantly change the content of our analysis. It seemed more significant to us that no information on the number of tests removed from the list by March 2021 is available for tests listed within the first six weeks since the regulation was introduced in October 2020.

Survey results

As of 17.11.2020, the BfArM list comprised 165 reimbursable PoC-AgTs. The proportion of identical rapid tests or instructions for use with identical content was 47.3% (78 of 165). A further 10 documents were not freely available online, meaning the total number of instructions for use was reduced to a total of 77 different PoC-AgTs. The survey was initially limited to purely recording the availability of performance data, without technical evaluation.

Tables 1 and 2 summarise the requirements defined by the PEI and the RKI for which information was available in the freely accessible instructions for use. Following Annex I of the EU in vitro Diagnostics Directive 2017/46 [5], a distinction is made between clinical (Table 1) and analytical (Table 2) performance characteristics. All providers stated a diagnostic sensitivity of >70% or a specificity of >97% for their product and were thus considered usable and reimbursable in the sense of the TestV. The sensitivity requirements were raised to 80% by the PEI in December 2020 and would also have been met by all products. However, looking at the lower limit of the 95% confidence interval, 23% (20 out of 87) already fell below the 80% mark according to the provider. The lower confidence interval for the minimum criterion of 97% specificity was undershot by 47% (41 of 87) of all PoC-AgTs. A total of eight of these 41 tests nevertheless achieved a specificity of 100%, but the lower limit of the confidence interval ranged from 96.7% down to 66%. Even currently (as of 02.03.2021), 16 (sensitivity) and 79 (specificity) different products whose 95% confidence limits lie outside the current minimum criteria are listed. Wide confidence intervals result from low numbers of cases and/or high variability of the tested variables. For the above-mentioned findings from the current list of the BfArM, in many cases the sample sizes were well above the required minimum of n≥100.

Table 1:

Availability of information on clinical performance characteristics in the test provider’s instructions for use.

Table 2:

Availability of information on analytical performance characteristics in the test provider’s instructions for use.

Figure 1 summarises the performance specifications for test sensitivity and specificity as well as the case numbers. A correlation between diagnostic and analytical sensitivity was not evident. The range of analytical sensitivity (detection limit) spanned four log levels.

Figure 1:

Graphical representation of clinical performance data of non-identical SARS-CoV-2 rapid tests (n=87).

Data on diagnostic sensitivity (●, red circles) and specificity (○, black circles) were taken from the list of applicable SARS-CoV-2 antigen rapid tests of the BfArM [2] (as of 17.11.2020). Provider information on the size of the tested collectives (■, blue bars) and the detection limit (◆, green diamonds) were available for 77 and 46 SARS-CoV-2 antigen rapid tests, respectively. The dashed vertical lines correspond to the default values [11] for diagnostic sensitivity (red), specificity (black), and sample size (blue). Vertical lines represent the 95% confidence intervals determined by the provider. Provider’s data on confidence intervals that did not appear plausible after visual inspection of the graph were recalculated. The corrected values are shown in the Figure. Areas highlighted in grey indicate test ID’s that were removed from the list after the closing date of 17.11.2020.

The comparability of detection limits (given by the manufacturers either as Ct values, as protein amounts in ng/mL or as TCID50) was limited by the following issue: Direct conversion of these different values is not readily possible. For this, detailed information on, for example, the qRT-PCR used, the viral strain or the recombinant protein would be required. In addition, for tests that detect spike protein, the detection limits may vary between different virus variants.

Figure 1 also illustrates that between November 2020 and March 2021, 25 (29%) of the 87 non-identical PoC-AgTs present on the cut-off date were removed from the list, with no apparent link to the test specifications.

The PEI dossier contains further important information on the reliable determination of analytical and clinical performance data. The instructions for use currently available from the test providers were therefore additionally evaluated with regard to these criteria. Two-thirds of all instructions for use did not contain any information on the study design and more than 80% did not contain any clinical information (e.g. symptomatic/asymptomatic). Thus, neither the conformity with the boundary conditions for the determination of clinical performance could be reliably checked, nor can performance data be compared between different providers. On a positive note, the validation reports of some test providers are freely accessible, including some raw data.

The Ct values of the RT-qPCR collected during test validation allow an orienting estimation of the virus concentrations. Since virus concentration and Ct values can vary considerably between different RT-qPCR methods, the information for use should also include meaningful information on the molecular diagnostic comparison method used. It is remarkable that this information was only available in 13% (10 of 77) of all leaflets, despite its relevance. Thus, unfortunately, all validation data based on Ct values, if available, can only be used to a limited extent.

Pre-analytical factors, such as the type and quality of the swab or transport medium, can also have a considerable influence on the determination of clinical performance data. It is therefore obvious that sufficient sample size, transparent description of the study design (e.g. prospective/retrospective or fresh swabs/frozen reserve samples) and the availability of clinical patient information or the characterisation of the samples used are quality criteria to be considered in any critical review of a PoC-AgT before its selection and application.

Since December 2020, the extension of the minimum criteria to include studies on cross-reactivity and interference has taken effect. Both performance characteristics allow conclusions to be drawn about the frequency of false-positive and false-negative results. Most of the instructions for use contain information on the pathogens tested and potential interferents. However, it is noticeable that more detailed data on the level of tolerated virus concentrations are available much less frequently, possibly due to a lack of test preparations.

Discussion

The German healthcare system is currently facing the challenge of compensating for diagnostic supply bottlenecks with alternative test concepts. Antibody testing is insensitive in the first week after infection and may serve as a complementary tool for individuals in a later disease state, when RT-PCR tests are negative, or not done [6]. The above-mentioned deficits of PoC-AgTs, which are already evident at the level of the performance data provided, make it clear that the push for high and rapid availability of such antigen tests is currently accompanied by potential quality deficits. All PoC-AgTs evaluated in this study are CE marked and should therefore meet basic minimum requirements for analytical and clinical performance. However, given the scope of SARS-CoV-2 diagnostics, it seems logical to further tighten the quality requirements for Coronavirus PoC-AgTs.

For various reasons, the BfArM and PoC-AgT users are faced with the challenge of monitoring the performance of PoC-AgTs on the market (post-market surveillance). The increasing number of publicly available, independent performance evaluation studies [7], [8] and meta-analyses [9], result in new options for the evaluation of PoC-AgTs. The recently published comparative evaluation [10] using a standardised set of clinical samples and their continuous update [11] complement the spectrum of independent sources of information.

It is becoming apparent that many manufacturers’ claims cannot be confirmed to the same extent in investigations at independent testing laboratories and that a market cleansing would be necessary. However, PoC-AgT users, especially larger health institutions, must understandably continue to look ahead to ensure that the selected product is of sufficient quality and sustainable when making larger procurements. From the DGKL’s point of view, when deciding on a PoC-AgT, users should give preference to those products that contain reliable and, above all, complete information on the minimum criteria in their instructions for use or via suitable other sources. All stakeholders need access to diagnostic performance data that have been collected under conditions that are as realistic as possible and that can be related to the current National Test Strategy for SARS-CoV-2. This is also necessary because the data situation for asymptomatic or pre-symptomatic patients is still insufficient. This includes all necessary additional information and in particular the description of the boundary conditions under which clinical performance data were collected (“should” requirements of the PEI). Otherwise, there is a risk that currently stocked PoC-AgTs could be removed from the list of applicable rapid antigen tests at a later date.

Reports on batch-dependent quality fluctuations also indicate the need to test adequate control samples at least before introducing the analysis and when changing a batch. Corresponding additional reagents are only offered by some test providers. The availability of control material should therefore also be taken into account when deciding on a test system.

Irrespective of all the above-mentioned aspects, sample collection and further pre-analytical principles, such as those mentioned below, are decisive for the quality of a PoC-AgT result [12] (Table 3). These measures not only improve the diagnostic quality but are also necessary in the context of occupational safety [13]. While some test provider information indicates that samples should not be inactivated, other providers offer sample inactivation procedures. The German Committee on Biological Agents (ABAS) recommends preferably working with inactivated samples or closed systems for the use of rapid tests in point-of-care facilities [13]. Some rapid tests are also suitable for test materials such as serum, which pose a lower risk of infection according to the ABAS. These rapid tests may therefore be an alternative for some facilities in unfavourable environmental conditions. All these aspects should ideally be clarified in advance of the decision for a PoC-AgT system on the basis of the provider’s information and underline the importance of comprehensive, high-quality documentation by the test provider.

Another chapter is now being opened by the availability and use of rapid antigen tests for use by medical laypersons (so-called “self-testing”) [14]. These products usually do not differ in their test configuration from the PoCAgTs already performed by medical professionals. However, pre-analytics and exact sample collection are of particular importance here. The gold standard here is still the deep nasopharyngeal swab. It is to be welcomed that, especially in the light of the planned mass use of this rapid antigen test, alternative sampling methods are evaluated, such as anterior nasal swab, saliva samples, etc. However, careful and broad evaluation of these alternative sample collection methods is required, together with transparent presentation of the results, so that an assessment of the test validity is also possible taking this additional dimension into account.

Conclusions

• The list compiled by BfArM and PEI comprises the available spectrum of approved, i.e. reimbursable, SARS-CoV-2 antigen rapid tests according to Corona-TestV.

• When making decisions regarding the selection and use of these products, users should consider test provider-independent sources of information in addition to the extended quality criteria of BfArM and PEI.

• Market cleansing should be supported by an appropriate acknowledgement of missing or insufficient performance data in the instructions for use.

• The correct collection of the samples contributes significantly to the quality of the results. This also applies to regulations on training, test execution and documentation of results that are adapted to the respective organisational, spatial and personnel conditions.

• Applicable recommendations of the ABAS on occupational health and safety measures and the individually existing environmental conditions should already be taken into account when selecting the product.

• Training of test users on the interpretation of results, especially on the limited significance of negative or positive test results, on the need for confirmation by PCR and on the temporary validity of the findings is required.

• Self-testing with rapid antigen tests by medical laypersons requires explicit instructions on pre-analytics, correct performance of the tests and on the correct behaviour according to the respective test result.