https://orcid.org/0000-0002-7990-1623
Figures
Abstract
Introduction
The Australian National COVID-19 Clinical Evidence Taskforce is producing living, evidence-based, national guidelines for treatment of people with COVID-19 which are updated each week. To continually improve the process and outputs of the Taskforce, and inform future living guideline development, we undertook a concurrent process evaluation examining Taskforce activities and experience of team members and stakeholders during the first 5 months of the project.
Methods
The mixed-methods process evaluation consisted of activity and progress audits, an online survey of all Taskforce participants; and semi-structured interviews with key contributors. Data were collected through five, prospective 4-weekly timepoints (beginning first week of May 2020) and three, fortnightly retrospective timepoints (March 23, April 6 and 20). We collected and analysed quantitative and qualitative data.
Results
An updated version of the guidelines was successfully published every week during the process evaluation. The Taskforce formed in March 2020, with a nominal start date of March 23. The first version of the guideline was published two weeks later and included 10 recommendations. By August 24, in the final round of the process evaluation, the team of 11 staff, working with seven guideline panels and over 200 health decision-makers, had developed 66 recommendations addressing 58 topics. The Taskforce website had received over 200,000 page views. Satisfaction with the work of the Taskforce remained very high (>90% extremely or somewhat satisfied) throughout. Several key strengths, challenges and methods questions for the work of the Taskforce were identified.
Conclusions
In just over 5 months of activity, the National COVID-19 Clinical Evidence Taskforce published 20 weekly updates to the evidence-based national treatment guidelines for COVID-19. This process evaluation identified several factors that enabled this achievement (e.g. an extant skill base in evidence review and convening), along with challenges that needed to be overcome (e.g. managing workloads, structure and governance) and methods questions (pace of updating, and thresholds for inclusion of evidence) which may be useful considerations for other living guidelines projects. An impact evaluation is also being conducted separately to examine awareness, acceptance and use of the guidelines.
Citation: Turner T, Elliott J, Tendal B, Vogel JP, Norris S, Tate R, et al. (2022) The Australian living guidelines for the clinical care of people with COVID-19: What worked, what didn’t and why, a mixed methods process evaluation. PLoS ONE 17(1): e0261479. https://doi.org/10.1371/journal.pone.0261479
Editor: Anna Ugalde, Deakin University, AUSTRALIA
Received: May 27, 2021; Accepted: December 2, 2021; Published: January 7, 2022
Copyright: © 2022 Turner et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying this study are publicly available at https://bridges.monash.edu/articles/dataset/National_COVID19_Clinical_Evidence_Taskforce_Process_Evaluation_Data/16926868.
Funding: The National COVID-19 Clinical Evidence Taskforce (TT, JE, BT, JV, SN, RT, SG) is funded by the Australian Government Department of Health - www.health.gov.au (MRF9200001), the Victorian Government Department of Health and Human Services(330251410), the Ian Potter Foundation - www.ianpotter.org.au (31110558) and the Walter Cottman Endowment Fund - www.eqt.com.au (EQTD01), managed by Equity Trustees, and the Lord Mayors’ Charitable Foundation - www.lmcf.org.au (20/1485). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Living guidelines are a new approach to developing and maintaining rigorous evidence-based guidelines in which any new evidence is rapidly incorporated, ensuring that recommendations are continually up to date with the latest research [1]. Living guideline methods are particularly useful in clinical areas in which research and practice are rapidly developing, of which COVID-19 is a clear example.
A small but increasing number of living guideline projects are underway, including in stroke, diabetes, maternal health and the recent living WHO guideline on drugs for COVID-19 [2–5], however little is currently known about what characteristics of the design or conduct of these projects make them more or less likely to succeed.
The Australian National COVID-19 Clinical Evidence Taskforce is a consortium of 32 Australian health professional organisations representing the full range of health professionals providing care to Australians with COVID-19, co-funded by Australian national and state governments and philanthropic organisations. The Taskforce is developing living guidelines for care of people with suspected or confirmed COVID-19 which are updated each week to reflect new evidence. The structure and methods used by the Taskforce to develop the guidelines have been described previously [6]. In brief, the guidelines use the rigorous GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach [7] and are designed to meet Australian National Health and Medical Research Council (NHMRC) standards [8]. Each week, the team identify and review new evidence; convene multiple multidisciplinary guideline panels; revise existing recommendations and make new recommendations. The guideline recommendations are published and freely accessible online in the MAGICapp online guideline platform, and are disseminated widely through mainstream and social media, and promoted widely by Taskforce member organisations [9].
Given the novel approach, unique opportunity to study rapid and living guideline development in new disease and the need to continuously improve processes in this rapidly evolving guideline, we undertook a process evaluation exploring the activity and experience of participants in the Taskforce for the first 5 months of the living guidelines project. The aim of the process evaluation was two-fold: to enable us to improve process and outputs of the Taskforce as guideline development was underway, and to identify factors that might be useful to inform design and development of future living guidelines, by capturing the activities and experiences of Taskforce contributors each month during the living guidelines project.
Methods
This process evaluation utilised a mixed methods approach consisting of activity and progress audits, online surveys, and semi-structured interviews. A protocol was developed for the process evaluation by TT. The protocol was refined in discussion with the Taskforce Executive team and approved by the Taskforce Steering Committee. Ethics approval was provided by Monash University Human Research Ethics Committee (Project ID: 24536).
Data were collected through five, prospective 4-weekly timepoints (beginning first week of May) and three, fortnightly retrospective timepoints (March 23, April 6 and 20). At each 4-weekly prospective timepoint we conducted an online survey of all participants in the Taskforce, as well as semi-structured interviews with key contributors and an audit of activity and progress. We collected and analysed both quantitative and qualitative data, taking a pragmatic approach focusing on rapidly informing Taskforce decision-making. For retrospective timepoints, we collected and analysed quantitative activity data only. The data underlying the results presented in this study are available on the Monash University Data Repository–BRIDGES (https://bridges.monash.edu/articles/dataset/National_COVID19_Clinical_Evidence_Taskforce_Process_Evaluation_Data/16926868).
Activity audit
Every 4 weeks, data were collected on a set of markers of the activity and progress of the Taskforce, outlined in Box 1. Data were collected from administrative reports (including Google Analytics), individual team members and MAGIC.
Box 1. Data collected in the activity audit
Number of recommendations made in guidelineNumber of clinical topics addressed by guideline
Number of citations screened by Evidence Team
Number of references included in guideline
Number of guideline panels convened
Number of guideline panel meetings conducted
Number of Slack messages sent by Evidence Team
Number of staff employed
Number of individuals engaged in Taskforce panels, committees, groups, etc
Number of Taskforce member organisations
Number of representative members in the Jurisdictional Liaison Group
Number of Taskforce funders
Number of Taskforce partners
Number of views of the Taskforce webpage Proportion of Taskforce webpage views from Australia
Number of Taskforce media mentions
Number Australian COVID-19 cases
Number of Australian COVID-19 deaths
Number of COVID-19 trials on ClinicalTrials.gov
Online survey
All members of Taskforce Executive, Steering Committee, Guideline Leadership Group, Guideline and Consumer Panels and the Evidence Team were invited to complete an online survey via direct email, sent every four weeks. This included both paid staff and voluntary contributors to the Taskforce (mostly clinicians). The survey was open for a week in each cycle. Participation was anonymous and voluntary, and completion of the online survey after reading the explanatory statement was considered implied consent.
Both quantitative and qualitative data were collected using an online survey tool (Qualtrix). Participants were asked to describe:
- Overall level of satisfaction with the work of the National COVID-19 Clinical Evidence Taskforce (5-point Likert: Extremely satisfied, Somewhat satisfied, Neither satisfied nor dissatisfied, Somewhat dissatisfied, Extremely dissatisfied)
- What is working well? (Free text)
- What could be improved? (Free text)
- With the benefit of hindsight, what would you do differently next time? (Free text)
Quantitative data was analysed using simple descriptive statistics, qualitative (free text) data were analysed in combination with the interview data.
Interviews
Interview participants included the Taskforce Executive Team, and purposively selected individuals with other roles in the Taskforce including clinical content experts and members of the evidence synthesis team. Taskforce members have a diverse range of roles. The Executive Team is comprised of clinicians, methodology experts, researchers, and the Taskforce communications and engagement manager, evidence and methodology manager, business manager and administration staff. Evidence team members largely have health science backgrounds and experience in evidence synthesis methodologies. Participants were invited to participate via direct email which included the explanatory statement. Participation was voluntary and agreeing to conduct the interview was considered consent. Five to ten semi-structured interviews were conducted with key members of the Executive Team and different Taskforce members during each of the five, four-weekly data collection cycles. Interviews were conducted via Zoom and were audio-recorded. Detailed notes were also taken. Interview questions were based on a predetermined interview schedule, with questions varied to suit the interviewee’s roles and experience. The interview schedule was developed by one author (TT) in consultation with the other authors.
Data were collected on:
- Satisfaction with progress, what was working well, which areas needed improvement
- Challenges, barriers, facilitators and enablers encountered
- What they might do differently with the benefit of hindsight
- Expectations for the next month
- What one thing would make the greatest difference to the work of the Taskforce
Interviews were conducted by one author (TT) who has extensive experience in qualitative interviewing, and data were de-identified. TT was known to the interviewees, some of whom are long-standing colleagues, and some of whom have been introduced during the work of the Taskforce. None of the participants were managed by or in a reporting relationship with TT. Each month, the interview data were combined with qualitative data from the survey, and were analysed using open coding to identify key concepts which were organised into emerging themes. TT undertook the primary data analysis, SG collaborated on the conceptual development and refining of themes. For this analysis, we primarily reviewed the reports to identify themes across time, but also revisited the primary data as necessary to confirm interpretation and review coding.
Results
Activity audits
The Taskforce formed in late March 2020, with a nominal start date of March 23. The first version of the guideline was published two weeks later and included 10, initially primarily consensus, recommendations. Using the rigorous, evidence-based methods described in detail elsewhere [6], the guidelines have been revised, updated and republished each week since.
Levels of activity were very high in the first month of Taskforce. The focus was on securing funding, rapidly establishing the team, convening three guideline panels covering mild, moderate-severe and critical COVID-19, engaging 20 partner organisations and establishing fundamental governance structures.
In May and June, the Taskforce built on this initial base to establish clear operational workflows and processes, expanded to include six guideline panels (addressing questions relevant to pregnant women, children, adolescents, older people, and people requiring palliative care); further developed engagement with jurisdictional decision makers (Commonwealth and state government groups) and strengthened the communications function.
By August 24, in the final round of the process evaluation, five months after establishment, the team consisted of 11 full-time equivalent staff, working with seven guideline panels and over 200 contributing individuals (health policymakers, practitioners and other stakeholders).
The evidence identified had informed development of 66 recommendations (version 18). The Taskforce website had received over 200,000 total pageviews and 116,383 unique visits.
Fig 1 provides a summary of the audit data over time. The complete data set is provided in Table 1.
Quantitative findings from online survey
The first survey was conducted from on May 4–11, seven weeks after the start of the Taskforce. The final survey was conducted from August 24–31, 23 weeks after the start of the Taskforce. The number of people invited to participate in the survey grew from 119 in May, to 207 in August; the response rate declined after the first two rounds, and as the number of individuals contributing to the Taskforce grew. Satisfaction with the work of the Taskforce remained very high (>85% extremely or somewhat satisfied) throughout (see Table 2).
Qualitative findings from online survey and interviews
A total of 38 interviews were conducted during the process evaluation, five in the first cycle, seven in the second, eight in the third, ten in the fourth and eight in the final cycle. Eight participants were interviewed multiple times (range: 2–5) and the other six participants were interviewed once. Interviewees included 7 members of the Executive Team, 6 members of the evidence synthesis team and a clinical content expert. Data from the interviews were combined with qualitative data collected through the online survey.
Strengths.
An updated version of the guidelines was successfully published every week during the process evaluation. This consistent delivery of the core output of the Taskforce led to pride and delight in the achievement of the Taskforce. Taskforce contributors were “astonished”, “amazed” and “impressed” by what was achieved and reported that they “feel privileged” to have been involved. “It’s amazing what we’ve achieved.”
Contributors repeatedly emphasised that a core strength of the Taskforce was that the team undertaking the evidence surveillance and living reviews were highly skilled, collegial and committed—“Trust and team work are extraordinary”. The core of the team was drawn from existing Cochrane Australia staff, rapidly bolstered by additional recruitment (often by secondment from within the Australian evidence synthesis community) during the start-up phase. Both having an initial team in place from which to build, and rapid recruitment of experienced individuals to expand this team, were major contributors to success.
In parallel with the expertise in evidence and guideline development methods, respondents regularly highlighted the extremely high levels of collaboration between Taskforce member organisations, and continuing high levels of engagement and contribution from individual clinicians. “The enthusiasm of the individual members is outstanding.”
While the early days of the Taskforce were retrospectively described as “frantic”, clear weekly workflows and processes were quickly established and continually refined. By May, respondents were reporting that the acute establishment phase had passed, and the Taskforce was developing a rhythm and structure; by late July team members reported that operational and evidence processes were running smoothly and efficiently.
Challenges.
Managing workload for the team was an ongoing challenge. Workloads during the establishment period were extremely high. While this settled a little as workflows were established and team members developed a “much clearer sense of what [they are] here to do”, for some topic areas such as drug treatments this was offset by an increasing rate of study publication. In the last round of the process evaluation there was recognition that “[we] still have people working ‘til midnight”.
The context of the COVID-19 pandemic created additional pressure and stress on the team and other Taskforce contributors, as many faced personal challenges, including increased clinical workloads; and most staff, based in Victoria, Australia, were in lockdown for many months, leading to mental and emotional exhaustion. “[W]e are all in a state of fatigue”. The context also meant it was not feasible for the team to meet face-to-face, and remote working was an additional challenge for communication and efficiency, particularly for new staff without existing relationships with Taskforce colleagues.
The structure of the team presented a challenge as the project progressed. In order to rapidly expand the team, many staff were seconded to the Taskforce for a finite period (most often 3–6 months), and although Taskforce funding was continued beyond this point, some had to return to their existing primary roles, creating skill gaps. Similarly, early expedient decisions to contract external providers for functions such as communication introduced challenges at later stages of the process.
Supporting guideline development panels was an ongoing challenge for the Taskforce. These panels of clinicians each met weekly to consider the evidence provided by the evidence team, and make or update the guideline recommendations. Providing timely, structured, concise but detailed pre- and post-meeting documentation was a significant undertaking. Discussions about the best size for these panels (smaller and more focused/efficient, versus broader and more representative, but less opportunity for discussion), and the scope of the portfolios of work addressed by each panel, continued throughout the period of the process evaluation. Some of the panels functioned effectively throughout, whereas others had periods of inefficiency and lack of clarity about contribution to the Taskforce. Highly skilled clinical chairs and methods co-chairs, were a vital contributor to the success of these panels.
Building on the initial success of the Taskforce, it was proposed the Taskforce expand scope to include infection prevention and control in healthcare settings. Those in favour of expansion saw an opportunity to address issues of vital concern to the guideline audience, however there were also concerns about whether expanding scope would put at risk the delivery of the primary output of the Taskforce, Australian living recommendations for the clinical care of people with COVID-19. These conversations continued throughout the process evaluation, and in the final round views were still mixed: “Exciting opportunities… logical extension”, “a bit scary in terms of resourcing”.
One of the components of the work of the Taskforce is to further build the technology platform to support living guideline production and publication. In the early phases of the work this was felt to be less of an issue, with respondents in June emphasising that “The limit is not tech, it’s humans”. However, the importance of the planned technological improvements was increasingly felt, and by the late August five months into the work, team members were increasingly aware of and frustrated by the tasks that they were doing manually, or things they were unable to do efficiently, because they were beyond the limits of available technology. “[The] tech stuff [is] still really challenging.”
One contributor to the delays in delivery of the technological development was siting of the Taskforce within the Monash University governance structure. There was recognition that Monash’s “admin set-up isn’t as agile as we are”. Location of the Taskforce within a university had important benefits including the provision of core financial and human resources infrastructure and governance oversight enabling rapid establishment, but also introduced significant challenges in the form of additional layers of bureaucracy and delays, especially in contracting. These challenges were exacerbated by the university-wide impacts of the COVID-19 pandemic, both on financial management, and increased administrative workload.
The all-pervasive uncertainty in which the work of the Taskforce was conducted, both in terms of the trajectory and impact of COVID-19 in Australia and globally, and also questions about the future funding and direction of the work of the Taskforce, became less comfortable for contributors over time. In the initial stages, contributors reported that they felt “privileged” to be involved and were altruistically motivated to contribute on an ad hoc basis. By July the Taskforce had reached “a tipping point” that “comes with maturity” where “people’s expectations are different” and there was increasing desire for a clear roadmap that described the future path of the Taskforce and expectations of ongoing contribution. There was also an increased desire for demonstration that the work of the Taskforce was having a meaningful impact. “I don’t have a good sense of whether we are really ‘landing’ with the target audience” “Are people actually using them?”.
Methods questions.
Two key issues about the living evidence synthesis methods employed in the guidelines were raised repeatedly throughout the process evaluation; the pace of guideline development, and the thresholds for inclusion of evidence.
From the beginning, the Taskforce committed to deliver weekly updates of the guidelines, a pace which was described as “superb”, “relentless” and “onerous”. To meet this aim, a weekly schedule was developed which included meetings of each of the guideline panels, as well as the Guideline Leadership Group to review recommendations, and the Steering Committee to approve recommendations. By June, some respondents were questioning whether weekly meetings were necessary and feasible for all guideline panels, or whether some panels might meet fortnightly or monthly without compromising the value of the guidelines, particularly in areas where less research was being published, or fewer new clinical questions raised. “[The] expectation that every week there will be something new is unrealistic.” Contributors had mixed feelings, with a sense that less frequent meetings were in some way devaluing their contribution, but also frustration if meetings didn’t seem to lead to concrete progress in the form of new recommendations. At the end of the process evaluation this question was still under discussion.
Given the context of a novel disease, with little prior research, initially all evidence was valuable to inform recommendations, and every new study was immediately appraised and incorporated. As the availability of research increased, a question arose about the value of the work required to rapidly incorporate results from small or low-quality studies into evidence summaries when it was clear that these would not impact on current recommendations. Some respondents suggested that an option was needed for a “holding bay”, or that thresholds should be set for incorporation of new evidence, perhaps similar to those used in living systematic reviews. Similarly, methods needed to be developed to deal with increasing numbers of studies available in preprint version.
Many of the key strengths and challenges identified through this process evaluation have application/relevance to other living guidelines (see Box 2). Regular, pre-determined updates are a significant strength of living guideline methodology. During times of acute need, such as a pandemic, weekly updates ensure the most current, evidence-based research is used to inform the rapidly evolving clinical landscape. Engaging highly skilled clinical and methods chairs and evidence team personal ensures efficiency of the panels and quality of the guidelines themselves.
Box 2. Key strengths and challenges of the living guideline methodology adopted by the Taskforce
Strengths
- *Weekly updates
- Highly skilled clinical chairs, methods co-chairs and evidence team
- Collaboration between member organisations
- High level of engagement of large numbers of clinicians
- *Clear, weekly workflows and processes
- University governance–rapid establishment, core human and financial resource infrastructure, governance oversight
Challenges
- *Workload management
- *Significant external, additional stressors on staff (mental and emotional exhaustion)
- Remote working
- *Rapid staff procurement
- Supporting the guideline development panels–ideal size, scope of portfolio of work to be addressed
- Inefficiencies with technology
- *University governance–contracting delays
- *Uncertainty–trajectory/impact of the pandemic and the guidelines
- Determining appropriate update schedule
- *Determining an appropriate, and potentially evolving threshold for inclusion of evidence
*Determining what to do with preprint studies
* Unique strengths/challenges during a pandemic.
Supporting guideline panels is likely to be a consistent challenge across living guidelines however the cumulative effect of supporting numerous members across several panels operating on a weekly update cycle is unique to the experience of the Taskforce. Inefficiencies with technology and determining an appropriate update schedule are challenges likely to be experienced across other living guidelines. Further consideration is required to provide clear recommendations for these areas.
Discussion
At the conclusion of the process evaluation, as of September 10, the living, evidence-based Australian guidelines for the clinical care of people with COVID-19 had been published and updated 20 times in just over 5 months, and included more than 70 recommendations, a significant achievement for one of the world’s first living guidelines, and certainly the most rapidly updated living guideline to date. Conducting a process evaluation alongside the work of the Taskforce was an important opportunity not only to improve Taskforce processes and outputs as the project progressed, but also to reflect on what future living guidelines and evidence synthesis projects might learn from this experience.
The work of the Taskforce was made possible by the existence of the team at Cochrane Australia with skills in evidence synthesis and guideline development methods, and the Australian Living Evidence Consortium which has been pioneering and piloting methods and platforms for living evidence synthesis [10]. Having this team in place allowed the Taskforce to rapidly ‘spin-up’, providing a core set of skills and leadership which formed the backbone of the team, which was then strengthened over time by recruitment of high-calibre individuals to fill specific skill gaps. This evaluation suggests core skills in evidence synthesis and GRADE guideline development are important determinants of success and would need to be available for other similar projects to progress rapidly.
The second strand of expertise that enabled the Taskforce was convening power. Guideline development relies on the integration of both evidence, consumer and clinical expertise, and the ability to engage and support over 200 contributors and manage their diverse, expert input each week was an important contributor to the credibility of the guidelines. Furthermore, the ability to convene more than 30 organisations representing health practitioners providing care to people with COVID-19 across Australia and to achieve 100% consensus across all these organisations was critical to the position of the guidelines as Australia’s ‘one-stop shop’ for COVID-19 clinical care.
The questions about pace of guideline updating and thresholds for inclusion are important considerations for future living guidelines projects. As in all projects, there is a trade-off between quality, time and resources at the heart of these decisions, and they are also likely to be dependent on the context and the clinical topic under consideration. Decisions about how often new evidence should be considered should be driven by the rate of emergence of new evidence, clinical uncertainty and importance of the area under review. The frequency with which panels are convened does not have to be directly reflected in the publication schedule for the guideline. A guideline with multiple guideline panels like those convened by the Taskforce can, for example, publish weekly updates with only some panels meeting each week. Indeed, this is the model that the Taskforce has now adopted, reflecting the transition from a focus on primarily developing new recommendations for many questions to the less-resource intensive work of maintaining recommendations in living mode.
Existing guidance for living systematic reviews [11] suggests that a priori decisions about thresholds for inclusion of new evidence should be made on the basis of whether the new evidence is likely to change the direction, clinical importance or certainty of the effect. Similar criteria could be useful for decisions about when to include new evidence in evidence summaries for living guideline recommendations, however it will be important to determine how feasible it is to develop these thresholds, how acceptable they are to guideline users and what effect they might have on workload and feasibility. Similarly, there are additional considerations about how to prioritise and select new questions to be addressed by the guidelines over time, and how and when to retire questions from living updates.
The recent rise in publication of living guidelines [2–5] and living systematic reviews [12–22], accelerated by the COVID-19 pandemic, makes addressing these methodological questions increasingly important. This growing body of living evidence syntheses also increases the availability of information about which approaches work best and in which contexts and clinical topic areas, which will be an important contribution to research underway to map these methods [23] and to further development of guidance for living guidelines [1] and other evidence syntheses [11, 24].
This study has some limitations, mostly arising from its pragmatic purpose to inform the conduct of the Taskforce as it was underway. We conducted only a small number of interviews in each round of the process evaluation, however these included all the senior leadership team in each cycle with the exception of two occasions when staff were on leave; along with a selection of other contributors. All Taskforce participants were invited to respond to the survey, and although participation declined over time, numbers were consistently above 25 each cycle, even during school holidays and Melbourne’s second wave of COVID-19. The combination of the regular interviews with the quantitative and qualitative data from the broader survey provides useful insights into the progress and mindset of the Taskforce at each timepoint.
A preliminary analysis of the results of each cycle of the process evaluation were reported to the Taskforce Executive and Steering Committee every four weeks, approximately one week after the data was collected and were used to inform and guide a process of continuous improvement of Taskforce processes and outputs. Providing the results rapidly and contemporaneously meant they could be used iteratively to identify and address emerging issues, and ascertain whether previously raised issues were being effectively addressed, while the project was underway. Some examples of use of the process evaluation data included work to refine roles within the Evidence Team and Executive to improve clarity and workflows; processes to restructure guideline panels to optimise efficiency; creation of deputy chair and methods co-chair roles in each panel to ensure appropriate levels of support; and development of governance mechanisms and standard operating procedures in response to issues raised. The rapid cycle of receipt of feedback through the process evaluation, development of responses by the Taskforce Executive and Steering Committee, and request for further feedback from the next timepoint of the process evaluation, also built confidence that the process evaluation was meaningful, that feedback was being heard, and that contributors’ views were valued. One unexpected benefit of the process evaluation interviews was the opportunity provided to interviewees for a time set aside to pause and reflect, and they felt that the opportunity to step away from the business of their day to day work was helpful both personally and professionally. Living guideline developers should consider establishing a similar model of process evaluation to inform the development and implementation of their guidelines.
Several factors should be considered when interpreting our findings. The most important of these is the context of a global pandemic in which the Taskforce has operated, which has had multiple impacts. The high profile and vital importance of COVID-19 as a global and national health threat was likely a major motivator for the high-levels of engagement in the work of the Taskforce, both for the member organisations and individual clinicians contributing to guideline panels, and for the Taskforce team. It is likely that this, and the resulting urgent demand for recommendations to guide clinical practice, provided an enabling environment for the high levels of collaboration and cooperation seen in the Taskforce. The COVID-19 context also placed additional pressure and stress on the team and Taskforce contributors, as many faced significant additional personal and professional challenges, including an extended lockdown period for staff based in Victoria. Remote working was the norm for the Taskforce, with staff working from home—every team, panel and group meeting was conducted in an online environment. These contextual factors perhaps limit generalisability of some of our findings, but the global pandemic also provides a useful stress-test of the living evidence model, demonstrating that it is possible to develop rigorous, evidence-based living guidelines even during times of crisis. This process evaluation presents useful learnings on the rapid development and establishment of living guidelines during a pandemic. We have demonstrated that rapid development of living GRADE-based guidelines is both feasible and acceptable and provided some insights to the strengths and challenges of living guideline methods in general.
Conclusion
The Australian Guidelines for the Clinical Care of People with COVID-19 are an important example of intensive application of living guideline methods. Despite the challenges of implementing a novel approach to evidence-based guideline development remotely during a pandemic, the Taskforce has demonstrated the feasibility of very frequent updating for topics in which stakeholders value very high levels of guideline currency.
The National COVID-19 Clinical Evidence Taskforce process evaluation provided useful data to improve the guidelines and the processes by which they were produced; and identified important considerations for future living guideline projects. An impact assessment is also being conducted to examine the extent of awareness, acceptance and use of the guidelines.
Acknowledgments
Our thanks to all of the members of the National COVID-19 Clinical Evidence Taskforce for their magnificent contributions to the work described in this manuscript. Steering Committee: Sharon McGowan (Chair), Nicola Ballenden, Terri-Lee Barrett, Vanessa Beavis, James Beckford Saunders, Tanya Buchanan, Marina Buchanan-Grey, Dawn Casey, Marita Cowie, Joseph Doyle, Mark Frydenberg, Danijela Gnjidic, Sally Green, Rohan Greenland, Ken Griffin, Stephan Groombridge, Louise Hardy, Alison Hodak, Anthony Holley, Vase Jovanovska, Sabina Knight, Kristin Michaels, Peter Morley, Julia Morphet, Suzi Nou, Phillip Russo, Megan Sarson, Alan Young. Executive Team: Julian Elliott, Rhiannon Tate, Britta Tendal, Sarah Norris, Bronwyn Morris-Donovan, Joshua Vogel, Sharon Gurry, Eloise Hudson, Shauna Hurley, Declan Primmer, Samantha Timms, Susan Whicker. National Guidelines Leadership Group: Julian Elliott (Co-Chair), Sutapa Mukherjee (Co-Chair), Joshua Vogel (Deputy Chair), Jason Agostino, Karen Booth, Lucy Burr, Lyn Byers, Peter Cameron, Megan Cooper, Allen Cheng, Peter Fowler, Mark Frydenberg, Alan Glanville, Caroline Homer, Karin Leder, Steve McGloughlin, Brendan McMullan, Ewen McPhee, Brett Mitchell, Mark Morgan, Paul Myles, Chris O’Donnell, Michael Parr, Jane Phillips, Rebecca Randall, Wayne Varndell, Ian Whyte, Leeroy William. Consumer Panel: Rebecca Randall (Chair), Richard Brightwell, Lynda Condon, Amrita Deshpande, Adam Ehm, Monica Ferrie, Joanne Muller, Lara Pullin, Elizabeth Robinson, Adele Witt. Primary and Chronic Care Panel: Sarah Larkins (Co-Chair), Mark Morgan (Co-Chair), Georgina Taylor (Deputy Chair), Jason Agostino, Paul Burgess, Penny Burns, Lyn Byers, Kirsty Douglas, Ben Ewald, Dan Ewald, Dianna Fornasier, Sabina Knight, Carmel Nelson, Louis Peachey, David Peiris, Mieke van Driel, Lucie Walters, Ineke Weaver. Hospital and Acute Care Panel: Lucy Burr (Chair), Simon Hendel (Deputy Co-Chair), Kiran Shekar (Deputy Co-Chair), Bronwyn Avard, Kelly Cairns, Allan Glanville, Nicky Gilroy, Paul Myles, Robert O’Sullivan, Owen Robinson, Chantal Sharland, Sally McCarthy, Peter Wark. Critical Care Panel: Steve McGoughlin (Co-Chair), Priya Nair (Co-Chair), Carol Hodgson, (Deputy Chair), Melissa Ankravs, Craig French, Kim Hansen, Sue Huckson, Jon Iredell, Carrie Janerka, Rose Jaspers, Ed Litton, Stephen Macdonald, Sandra Peake, Ian Seppelt. Pregnancy and Perinatal Care Panel: Caroline Homer (Co-Chair), Vijay Roach (Co-Chair), Michelle Giles (Deputy Co-Chair), Clare Whitehead (Deputy Co-Chair), Wendy Burton, Teena Downton, Glenda Gleeson, Adrienne Gordon, Jenny Hunt, Jackie Kitschke, Nolan McDonnell, Philippa Middleton, Jeremy Oats. Paediatric and Adolescent Care Panel: Asha Bowen (Co-Chair), Brendan McMullan (Co-Chair), David Tingay (Deputy Co-Chair), Nan Vasilunas (Deputy Co-Chair), Lorraine Anderson, James Best, Penny Burns, Simon Craig, Simon Erickson, Nick Fancourt, Zoy Goff, Vimbai Kapuya, Catherine Keyte, Lorelle Malyon, Danielle Wurzel. Palliative and Aged Care Panel: Meera Agar (Co-Chair), Richard Lindley (Co-Chair), Natasha Smallwood (Deputy Chair), Mandy Callary, Michael Chapman, Philip Good, Peter Jenkin, Deidre Morgan, Vasi Naganathan, Velandai Srikanth, Penny Tuffin, Elizabeth Whiting, Leeroy William, Patsy Yates. Disease-Modifying Treatment and Chemoprophylaxis Panel: Bridget Barber, Jane Davies, Josh Davis, Dan Ewald, Michelle Giles, Amanda Gwee, Karin Leder, Gail Matthews, James McMahon, Trisha Peel, Chris Raftery, Megan Rees, Jason Roberts, Ian Seppelt, Tom Snelling, Brad Wibrow. Expert Advisory Group: Ross Baker, Jennifer Curnow, Briony Cutts, Anoop Enjeti, Andrew Forbes, Prahlad Ho, Adam Holyoak, Helen Liley, James McFadyen, Zoe McQuilten, Eileen Merriman, Helen Savoia, Chee Wee Tan, Huyen Tran, Chris Ward, Katrina Williams. Cardiac Arrest Working Group: Neil Ballard, Samantha Bendall, Neel Bhanderi, Lyn Byers, Simon Craig, Dan Ellis, Dan Ewald, Craig Fairley, Brett Hoggard, Minh Le Cong, Peter Morley, Priya Nair, Andrew Pearce. Evidence Team: Britta Tendal, Steve McDonald, Tari Turner, Joshua Vogel, David Fraile Navarro, Heath White, Samantha Chakraborty, Saskia Cheyne, Henriette Callesen, Sue Campbell, Jenny Ring, Agnes Wilson, Tanya Millard, Melissa Murano. Observational Data Working Group: David Henry (Co-Chair), Sallie Pearson (Co-Chair), Douglas Boyle, Kendal Chidwick, Wendy Chapman, Craig French, Chris Pearce, Tom Snelling. Independent Conflicts of Interest Committee: Lisa Bero (Chair), Quinn Grundy, Joel Lexchin, Barbara Mintzes.
We would like to acknowledge the Member Organisations: Australian Living Evidence Consortium (Convenor), Cochrane Australia (Secretariat), Australasian Association of Academic Primary Care, Australian Association of Gerontology, Australasia College for Emergency Medicine, Australasian College for Infection Prevention and Control, Australasian Society for Infectious Diseases, Australasian Society of Clinical and Experimental Pharmacologists and Toxicologists, Australian and New Zealand College of Anaesthetists, Australian and New Zealand Intensive Care Society, Australian & New Zealand Society for Geriatric Medicine, Australian College of Critical Care Nurses, Australian College of Midwives, Australian College of Nursing, Australian College of Rural and Remote Medicine, Australian COVID-19 Palliative Care Working Group, Australian Primary Health Care Nurses Association, Australian Resuscitation Council, Australasian Sleep Association, Australian Society of Anaesthetists, Australasian College of Paramedicine, College of Emergency Nurses Australasia, CRANAplus, National Aboriginal Community Controlled Health Organisation, Royal Australasian College of Physicians, Royal Australian College of Surgeons, Royal Australian and New Zealand College of Obstetricians and Gynaecologists, Royal Australian College of General Practitioners, Society of Hospital Pharmacists of Australia, Thoracic Society of Australia and New Zealand, Thrombosis and Haemostasis Society of Australia and New Zealand.
We would like to acknowledge our Partners: Australian Commission on Safety and Quality in Health Care, Cochrane, Consumers Health Forum of Australia, Covidence, Hereco, MAGIC, NPS MedicineWise, School of Public Health and Preventive Medicine, Monash University.
We would like to acknowledge Simon Turner for producing the graphs.
References
- 1. Akl EA, Meerpohl JJ, Elliott J, Kahale LA, Schunemann HJ, Living Systematic Review N. Living systematic reviews: 4. Living guideline recommendations. J Clin Epidemiol. 2017;91:47–53. pmid:28911999
- 2. White H, Tendal B, Elliott J, Turner T, Andrikopoulos S, Zoungas S. Breathing life into Australian diabetes clinical guidelines. Med J Aust. 2020;212(6):250–1 e1. pmid:32058591
- 3. English C, Bayley M, Hill K, Langhorne P, Molag M, Ranta A, et al. Bringing stroke clinical guidelines to life. Int J Stroke. 2019;14(4):337–9. pmid:30806591
- 4. Vogel JP, Dowswell T, Lewin S, Bonet M, Hampson L, Kellie F, et al. Developing and applying a ’living guidelines’ approach to WHO recommendations on maternal and perinatal health. BMJ Glob Health. 2019;4(4):e001683. pmid:31478014
- 5. Lamontagne F, Agoritsas T, Macdonald H, Leo YS, Diaz J, Agarwal A, et al. A living WHO guideline on drugs for covid-19. BMJ. 2020;370:m3379. pmid:32887691
- 6. Tendal B, Vogel JP, McDonald S, Norris S, Cumpston M, White H, et al. Weekly updates of national living evidence-based guidelines: Methods for the Australian Living Guidelines for Care of People with COVID-19. Journal of Clinical Epidemiology. pmid:33188858
- 7. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. 1GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices: Introduction. BMJ. 2016;353:i2016. pmid:27353417
- 8. Salyer K. COVID-19: What to know about the coronavirus pandemic on 6 April: World Economic Forum; 2020 [Available from: https://www.weforum.org/agenda/2020/04/covid-19-what-to-know-about-the-coronavirus-pandemic-on-6-april/.
- 9.
Making GRADE the Irresistible Choice (MAGIC) Project. 2018 [Available from: http://magicproject.org/.
- 10.
Australian Living Evidence Consortium 2020 [Available from: https://livingevidence.org.au/.
- 11. Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, et al. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30. pmid:28912002
- 12. Update to living systematic review on prediction models for diagnosis and prognosis of covid-19. BMJ. 2020;370:m2810. pmid:32694132
- 13. Andrenelli E, Negrini F, de Sire A, Arienti C, Patrini M, Negrini S, et al. Systematic rapid living review on rehabilitation needs due to COVID-19: update to May 31st, 2020. Eur J Phys Rehabil Med. 2020;56(4):508–14. pmid:32539312
- 14. Baladia E, Pizarro AB, Ortiz-Munoz L, Rada G. Vitamin C for COVID-19: A living systematic review. Medwave. 2020;20(6):e7978. pmid:32759894
- 15. de Sire A, Andrenelli E, Negrini F, Negrini S, Ceravolo MG. Systematic rapid living review on rehabilitation needs due to COVID-19: update as of April 30th, 2020. Eur J Phys Rehabil Med. 2020;56(3):354–60. pmid:32408729
- 16. Deeks JJ, Dinnes J, Takwoingi Y, Davenport C, Spijker R, Taylor-Phillips S, et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst Rev. 2020;6:CD013652. pmid:32584464
- 17. Hernandez AV, Roman YM, Pasupuleti V, Barboza JJ, White CM. Hydroxychloroquine or Chloroquine for Treatment or Prophylaxis of COVID-19: A Living Systematic Review. Ann Intern Med. 2020;173(4):287–96. pmid:32459529
- 18. Piechotta V, Chai KL, Valk SJ, Doree C, Monsef I, Wood EM, et al. Convalescent plasma or hyperimmune immunoglobulin for people with COVID-19: a living systematic review. Cochrane Database Syst Rev. 2020;7:CD013600. pmid:32648959
- 19. Schunemann HJ, Khabsa J, Solo K, Khamis AM, Brignardello-Petersen R, El-Harakeh A, et al. Ventilation Techniques and Risk for Transmission of Coronavirus Disease, Including COVID-19: A Living Systematic Review of Multiple Streams of Evidence. Ann Intern Med. 2020;173(3):204–16. pmid:32442035
- 20. Struyf T, Deeks JJ, Dinnes J, Takwoingi Y, Davenport C, Leeflang MM, et al. Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19 disease. Cochrane Database Syst Rev. 2020;7:CD013665. pmid:32633856
- 21. Thombs BD, Bonardi O, Rice DB, Boruff JT, Azar M, He C, et al. Curating evidence on mental health during COVID-19: A living systematic review. J Psychosom Res. 2020;133:110113. pmid:32354463
- 22. Verdugo-Paiva F, Izcovich A, Ragusa M, Rada G. Lopinavir-ritonavir for COVID-19: A living systematic review. Medwave. 2020;20(6):e7967. pmid:32678815
- 23. Khamis AM, Kahale LA, Pardo-Hernandez H, Schunemann HJ, Akl EA. Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey. F1000Res. 2019;8:221. pmid:31231512
- 24. Brooker J, Synnot A, McDonald S, Elliott J, Turner T. Guidance for the production and publication of Cochrane living systematic reviews: Cochrane Reviews in living mode. Version December 2019. 2019.