Covid-19: Evidence used to support mass lateral flow testing was flawed, experts say
BMJ 2021; 374 doi: https://doi.org/10.1136/bmj.n1907 (Published 28 July 2021) Cite this as: BMJ 2021;374:n1907Read our latest coverage of the coronavirus pandemic
Experts in medical testing have expressed concern at the way data were collected in key studies that formed the evidence base for the use of the Innova lateral flow devices in the NHS.
A paper analysing the devices’ diagnostic performance, published in the journal EClinicalMedicine on 29 May,1 concluded that they “have promising performance characteristics for mass population testing and can be used to identify infectious positive individuals.”
Innova’s lateral flow tests were used in pilots of community testing for SARS-CoV-2 in England and subsequently deployed for mass testing of asymptomatic people as part of the government’s costly and ambitious “Operation Moonshot” programme.234 This was done under an exceptional use authorisation (EUA) from the Medicines and Healthcare Products Regulatory Agency, which covers only NHS rebranded Innova lateral flow tests used in the national mass testing programme. The EUA runs until 28 August.
Innova’s device “shows good viral antigen detection/sensitivity with excellent specificity, although kit failure rates and the impact of training are potential issues,” said the paper, which was based on analyses by researchers at Public Health England’s Porton Down facility and the University of Oxford.
But in a letter to EClinicalMedicine published on 15 July,5 members of the Cochrane Covid-19 Diagnostic Test Accuracy Group said that there were discrepancies between the report of the studies that was published in the journal, information presented in the researchers’ preprint,6 and data that the investigators provided separately to the Cochrane team. It said that the Standards for Reporting of Diagnostic Accuracy Studies (STARD) criteria, adopted by the Lancet journals, “have not been fully adhered to by this article.”
“We have great concern that the evidence supporting the use of Innova for mass testing has not been obtained and reported with adequate attention to the scientific method required to provide a sound evidence base for policy-making,” the letter says.
The Cochrane team requested clarification from the paper’s authors on several specific points (box 1).
Cochrane group’s three concerns about the studies5
“1) The paper computes specificity by combining data on 6954 Innova tests across studies from ‘negative samples’ (Table 2). This implies all 6954 samples were verified by a reference standard which returned a negative result (anticipated to be RT-PCR, but no reference standard is described). The preliminary report indicated none of the 3985 participants in the school studies contributing to the estimate of specificity received RT-PCR or any other test to verify their disease status, and the data sent to the Cochrane team [to Jon Deeks] indicated that the 17 participants in the school studies who tested positive on the Innova assay were parallel-tested with RT-PCR. Can the authors clarify which students received a reference standard, and if so, what test was used and when it was done?
“If full verification of disease status was not performed, computing specificity using a sample size of 6954 (rather than 2969) is not correct. Notably only 1852 of these samples appear to have been tested using dry swabs in line with manufacturer's instructions, so the relevant sample size is likely even smaller.
“2) Of the 17 school samples reported to the Cochrane team as Innova positive, 14 were false positives as the RT-PCR test was reported negative. Whilst unable to compute sensitivity or specificity from these studies, knowing the numbers of test positives and false positives allows computation of a positive predictive value of 18% (95% confidence interval 4 to 43%). This important result is not reported in the publication or preprint. (The publication suggests 16 false positives and higher total numbers tested, so the exact value may slightly differ). The positive predictive value of school testing has been an important debate, particularly given school testing was originally introduced without RT-PCR confirmatory testing, thus its omission from this report is of relevance. Can the authors clarify that this result is correct and explain why it has been excluded from this report?
“3) The authors report a specificity of 100% but no sensitivity value for the Armed Forces cohort. A full 2×2 table for this study was provided to and included in the Cochrane review—Innova was positive in 13 out of 46 who were PCR positive (sensitivity 28, 95% confidence interval (16 to 43%)). Why has this result not been reported?”
RETURN TO TEXTOn 22 July Lennard Lee and Tim Peto from the Oxford research team issued a response to the concerns raised, which was also published as a letter in EClinicalMedicine (box 2).7
How the authors responded7
“The main evaluation of the lateral flow devices is reported in the Phase 3 studies. The Phase 4 studies were designed to determine the feasibility of using lateral flow devices in different settings. As the disease prevalence at the time of these studies was very low, the Phase 4 studies were not powered to undertake a standard prospective evaluation of both positive and negative cases.
“We are happy to confirm that all the samples shown in Table 2 had both a negative PCR test and a lateral flow test allowing the false positive rate to be estimated. PHE Porton tested the PCR samples collected from the school and armed forces studies. We agree that for the Phase 2 and 3 studies, lateral flow tests were done in laboratory conditions but overall 5942/6954 (85.4%) of the tests were done following the manufacturing instructions (rather than the 1852 suggested by the writers).
“The writers refer to a personal communication providing Prof Deeks preliminary with the results then available. The current publication provides a cleaned version of the results. The specificity results from the very small Armed Forces study was included as we were interested in the false positive rate, at a time of lower disease prevalence. We apologize for not including in this paper the sensitivity results which we sent directly to Prof Deeks for the Cochrane Review. These show that the numbers (pos/total) were 8/8 (CT < 25); 3/4 (CT > = 25 to 28) and 2/34 (CT > = 28). As the disease was waning in the population at the time, the prevalence of low viral loads is not surprising and the low positive rate in such a population is expected and fits with the more complete Phase 3a results presented in our paper.
“We do not consider that an overall ‘sensitivity’ of the test against a categorical PCR result, without accounting for viral load, is a useful measure of LFD performance as all studies confirm that the LFD test is normally negative in individuals with low viral loads. The author’s quoting of a sensitivity of only 28% and a positive predictive value of 18% at low disease prevalence is misleading, as already discussed as it assumes that the PCR test is a reliable test for infectiousness.”
RETURN TO TEXTIn addition to replying to the concerns about their data, Lee and Peto also defended the wide use, based on the evidence, of lateral flow devices in the UK and elsewhere.
“Many thousands of infectious individuals have been identified and subsequently isolated successfully, breaking the chain of transmission,” they wrote. “For many, the return to a semblance of normal life has been facilitated by lateral flow tests.”
They added, “There is no practical alternative as the PCR [polymerase chain reaction] test has poor specificity in identifying infectious cases, is inaccessible, expensive and has a slow turn-around rate.”
In a statement the Lancet group said, “The STARD guidelines help improve completeness and transparency in reporting of diagnostic accuracy studies. Post-publication, scientific discussion, and debate are an important part of the scientific process, and the Lancet journals welcome responses from the wider scientific community to content published in our journals.”
This article is made freely available for use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.
https://bmj.com/coronavirus/usage