1 Introduction

The response to the coronavirus disease 2019 (COVID-19) pandemic included a global effort to monitor the benefits and risks of vaccines and therapeutics, when used in the real world (i.e., ‘post-marketing’). This rapid and large-scale activity presented a new level of challenges, met with unprecedented levels of innovation, cooperation, and lay public communication. Similar challenges had been experienced in the 2009 H1N1 pandemic and, although some lessons learned triggered initiatives and solutions [1, 2] to improve response and cooperation among key stakeholders, there was insufficient continuity and coordination of pandemic preparedness between all stakeholders following H1N1. To improve pandemic preparedness following COVID-19, it is critical to keep the current momentum of cooperation among key stakeholders and to expand the conversation to other vaccines and to therapeutics, when relevant.

BeCOME was launched in 2022 to build on the achievements and learnings attained through multi-stakeholders' cooperative efforts in response to the unprecedented public health challenge posed by the COVID-19 pandemic (e.g., building the COronaVIrus Disease Research Initiative on Vaccine Effectiveness [COVIDRIVE] platform, subsequently expanded as id.DRIVE [3]). Having experienced the value of a cross-company collaborative approach [4] and realizing the extent to which possible solutions to key global challenges are joint efforts with health authorities, public institutions, and other organizations, the vision was to rapidly engage with multiple stakeholders (i.e., regulatory authorities, academic institutions, public health experts, research networks, and non-government organizations) to create a sustainable forum. Thus, experts in pharmacovigilance and pharmacoepidemiology could discuss, collaborate on, and develop new methods and initiatives regarding the real-world monitoring of benefits and risks of vaccines to improve preparedness and ultimately the response for future pandemics. To initiate development of a plan for achieving this vision, BeCOME hosted its first conference in June 2023 to present and discuss mutual priorities for the next 5 years. In turn, these mutual priorities were to inform cooperative action plans to improve joint preparedness across stakeholders.

2 Inaugural Cross-Stakeholder Meeting

“Beyond COVID-19: the Future of the Real-World Monitoring of Vaccines” was held June 11–13, 2023, at Les Pensières Center for Global Health in Annecy, France (see online supplementary material for conference program). Les Pensières, set aside by the Mérieux Foundation to host events and collaborations focused on public health, was a perfect setting for this gathering of 66 international experts from a broad spectrum of stakeholders (industry, academia, public health and academic research institutes, supra-national organizations, non-government organizations, regulatory agencies) and including attendees from low- and middle-income countries (LMICs). The meeting steering committee comprised representatives from industry (n = 3), the Centers for Disease Prevention and Control (n = 2), and the Bill & Melinda Gates Foundation (n = 1). Key stakeholders came together to identify and make recommendations on strategic priorities, solutions, timelines, and models for collaboration to expand on the cooperation initiated during the pandemic. The meeting began with an inspiring keynote address from Dame June Raine, CEO at the UK Medicines and Healthcare products Regulatory Agency, highlighting the importance of vaccine vigilance and the various challenges, innovations, and collaborations seen over the last 2 decades. Dr. Raine’s words served as a call to action for the BeCOME conference participants to build upon the learnings, technological advances, new data sources and capabilities, digital transformation, new communication methods to meet growing public expectations, and novel collaboration models to create a blueprint for future systems.

Opening remarks by Dr. Shanthi Pal, Team Lead for pharmacovigilance at the World Health Organization (WHO) and sitting board member at the Uppsala Monitoring Centre, also recognized the efforts that led to BeCOME. Dr. Pal urged conference participants to think broadly, avoid siloed systems, and—importantly—to consider both the needs of vulnerable settings and special populations and how any advances in safety surveillance may benefit medicines as well as vaccines.

Day 1 included sessions covering the following BeCOME areas of focus for post-COVID-19 vaccine monitoring: background incidence rates (BIRs), safety studies, pregnancy surveillance, vaccine benefits, signal detection, digital solutions, LMICs, and a transversal session on multi-stakeholder collaborations. During each session, the audience was oriented to multiple perspectives on major challenges and potential solutions under consideration, focusing on those relevant to a cooperative approach across multiple stakeholders. Each session concluded with an engaging panel discussion. Day 2 featured a series of 90-minute workshops for each area of focus, with the goal to gain alignment and reach consensus on priorities and recommendations for solutions and actions, followed by a final plenary defining key overall priorities and next steps.

3 Key Focus Areas and Identified Strategic Priorities

3.1 Background Incidence Rates

BIRs are critical data for interpreting signal monitoring and signal refinement efforts. Timely assessments of vaccine safety signals require comprehensive and valid BIRs of adverse events of special interest (AESI), especially in mass vaccination programs. Challenges faced by multiple stakeholders were reviewed:

  • During the COVID-19 pandemic, various lists of AESI emerged, each evolving over time. Furthermore, BIR could be dated, unavailable, or missing for some AESI. Developing a harmonized list of AESI that is applicable for all vaccines across stakeholders is mutually beneficial but presents challenges, as BIRs are specific to the targeted disease and vaccine platform. Nonetheless, activation of systems capable of promptly generating the most up-to-date BIR—for both established and novel AESI—in a format for efficient utilization is warranted.

  • The lack of harmonized methods for validation of case definitions or phenotypes in real-world data coupled with the heterogeneity across BIRs for a given AESI makes it challenging to conduct analyses and interpret results. The credibility of safety signals hinges on consistency across BIRs, which can be accomplished through standardization and validation of methodology and case definitions.

  • The absence of detailed BIRs brings difficulties in evaluating safety signals in different age groups, by gender, or in specific sub-populations such as pregnant individuals and the elderly. Additionally, BIRs are frequently unavailable for all countries of interest, especially LMICs. Generating granular-level or specific BIRs can improve the ability to assess safety signals [2].

To address these challenges, the following key strategic priorities were identified.

  1. 1.

    Establish a global multi-stakeholder working group to enable sustainable delivery of BIRs for a harmonized list of well-established and newly identified AESI.

  2. 2.

    Achieve multi-stakeholder agreement on best practices on the proper application of BIRs, addressing data quality, management of heterogeneity, and standardized reporting of data.

  3. 3.

    Improve the availability and granularity of data (i.e., by age and sex stratifications [both separately and dually stratified]; in specific sub-populations, including pregnant and lactating/breastfeeding people, high-risk groups, etc.; in specific geographic regions, including LMICs).

3.2 Safety Studies (Active Pharmacovigilance Surveillance)

There is a need for timely, high-quality, fit-for-use safety studies. Study design requirements must be balanced with study feasibility, data quality, and availability of results with public health urgency. Challenges specific to designing and conducting vaccine safety studies during a pandemic include the following.

  • Insufficient numbers of pharmacoepidemiologists or individuals trained in pharmacoepidemiology to design and execute safety studies.

  • Lack of visibility of data sources and study methods used for safety studies, including what data sources and methods were commonly used, what worked, and what did not work and under what circumstances.

  • Lack of shared understanding of what tools can be used, when, and how, across the continuum from passive surveillance to full safety studies.

  • Lack of a shared understanding of data source characteristics and operationalization of such sources globally, including LMICs.

  • Limited understanding of the surveillance infrastructure to conduct safety studies, especially in LMICs.

To further meet these and future challenges, the following key strategic priorities were identified.

  1. 1.

    Improve knowledge of commonly used data sources and associated data source characteristics for vaccine safety studies.

  2. 2.

    Increase global visibility of what data sources and study methods worked and under what circumstances to inform the areas of focus for future development and innovation.

  3. 3.

    Create shared understanding of what tools and/or methods can be used across the continuum of surveillance approaches (i.e., passive surveillance, observational studies); this includes promoting existing yet underused methods (i.e., through preparation of whitepaper and associated toolbox for safety surveillance methods). Tools may include artificial intelligence, intelligent automation, and machine learning that can be leveraged to respond to rapidly changing vaccine platforms. Such innovations are needed to identify and track vaccine platform changes and changes to product safety that may occur over time and ensure optimized use for translation of clinical case definitions into clinical phenotypes/surveillance case definitions.

  4. 4.

    Improve cross-stakeholder visibility of surveillance infrastructure globally, especially within LMICs.

3.3 Pregnancy Surveillance

The pregnancy surveillance workshop discussion centered around two main themes: (1) maintaining and improving the quality of studies, and (2) generating safety evidence faster and more efficiently. Challenges specific to studying vaccine exposure and safety in the pregnant population include the following.

  • Inconsistent guidance from regulatory authorities on the conduct of post-marketing safety studies in pregnant people that leads to inefficiency in study design, feasibility assessments, and—in turn—later delivery of actionable evidence to inform safety decision making.

  • Limitations of existing data sources to conduct post-approval pregnancy safety studies, including long data lags, limited ability to capture vaccine exposure information, and mother–infant linkage. Additional tools for characterizing and assessing data quality (i.e., relevance and reliability) are needed, as are standards to ensure the representativeness of the safety study population.

  • Limitations of pregnancy surveillance and pharmacovigilance systems in LMICs, including gaps in infrastructure, resources, training, data quality, and methods.

To address these challenges, the following key strategic priorities were identified.

  1. 1.

    Harmonization of regulatory guidance for the conduct of post-approval pregnancy safety studies by developing core safety study templates.

  2. 2.

    Development of standard and updated information on data sources that can meet regulatory requirements (including LMIC populations) and drive increased infrastructure to deliver faster safety evidence generation.

  3. 3.

    Expansion and improvement of pregnancy surveillance and pharmacovigilance systems in LMICs.

3.4 Vaccine Benefits

In BeCOME, the ‘vaccine benefit’ expert group broadly encompasses activities related to the benefit of vaccines, including measuring vaccine effectiveness, characterizing disease burden, and evaluating impact. Challenges specific to vaccine benefit evaluation include the following.

  • High levels of resources and expertise are needed for infectious disease surveillance and primary data collection activities that evaluate vaccine benefit.

  • Limited infrastructure is available to evaluate benefit, particularly in sub-populations and for rare diseases, where study power/precision of effectiveness estimates are problematic.

  • Speed of data access and delays in preparing valid outcome algorithms in many databases often delay rapid assessment and communication of benefit to decision makers.

  • Linkage between disease surveillance and vaccine registries is not universal, and key variables required for evaluation of vaccine benefit are often missing or unavailable in existing data sources.

  • Information required to evaluate vaccine benefit is often siloed by country, pathogen, and/or population.

  • Bias, particularly due to unmeasured confounding, remains a threat to the validity of estimates of vaccine effectiveness.

The following key strategic priorities for multi-stakeholder collaboration were identified as potential solutions to the specific challenges facing vaccine benefit stakeholders.

  1. 1.

    Leveraging and expanding existing data collection systems, platforms, and networks to (a) more efficiently and quickly conduct primary data collection studies with improved power, (b) enrich secondary data sources with multiplex polymerase chain reaction testing and vaccination/brand data as alternative ways to estimate effectiveness, (c) encourage open data for multi-stakeholders' research purposes, and (d) use partnerships to allow wider access to secondary data.

  2. 2.

    Increasing the speed of vaccine benefit analyses by considering the use of common data models, considering integration of data from multiple sources, obtaining more timely access to data and standardized analyses (thereby reducing need for primary data to leave sites, which in turn reduces bureaucracy), and developing detailed template protocols for evaluating vaccine effectiveness using different data sources and study designs.

  3. 3.

    Improving communication between stakeholders to identify priorities and agree on who can best deliver what information and by when. Specific to post-marketing studies of effectiveness, improved clarity is needed around responsibilities and ownership of activities with the aim of preventing duplicated work and effort.

  4. 4.

    Developing guidance on the use of real-world evidence specifically for decision making around vaccine benefit, with input sought from multiple stakeholders on the acceptability and value of different study designs, including novel applications of self-controlled risk interval for effectiveness; the use of carefully chosen negative exposures and outcomes to determine the existence of potential bias; and other advanced methods for bias understanding and adjustment.

  5. 5.

    Harmonize the implementation of benefit–risk studies by developing standardized methods and encouraging transparent reporting of detailed methodological and operational definitions.

3.5 Signal Detection

Participants in the signal detection workshop discussed the collective experience and potential value of traditional signal detection methods in conjunction with the observed-to-expected (O/E) analysis, mandated in many jurisdictions during the pandemic. Although the scope of signal detection activities has expanded over the past 2 decades to include new data sources, algorithms, and expertise across passive and active surveillance platforms, no studies have been conducted to determine what methods have the best predictive value, in a time- or exposure-dependent manner, across the pandemic timeline. During the pandemic, O/E analyses were required for signal generation/validation and characterization of safety data/topics in the requested safety monthly aggregate reports. This method was historically reserved for refinement of signals previously detected by traditional pharmacovigilance methods (index case on medical review, statistical, or other disproportionality analyses). To meet narrow reporting timelines, non-adjudicated cases were included as “observed” counts, which raised significant challenges for the interpretation of results. This method posed difficulties interpreting adverse events (AEs) that were also associated with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection circulating widely at the time. The heavily weighted O/E-related activities required multiple layers of stratification and impacted both AE and exposure data. Focusing on these time-consuming O/E activities diverted attention away from more traditional methods that require time for individual case scrutiny, exchange of follow-up information, and data maturation prior to the evaluation of evidence supporting or refuting a causal association with vaccine. Also, insufficient effort was devoted to analyzing the limitations of O/E studies and to promoting the use of alternative methodologies, including but not limited to self-controlled techniques with lower risk of bias. With this understanding, the following key strategic priorities were identified.

  • Identify signal detection data sources and methods applied by different stakeholders across the pandemic.

  • Assess performance of signal detection methods (including non-quantitative approaches such as clinical case review).

  • Discuss and propose consensual best practices for safety evaluation, including but not limited to the use of innovative methods to reduce bias, improving the use and validity of O/E methods, adjust for under-reporting of AEs, and the use of safety summary reports as a tool to communicate O/E and signal detection findings to health authorities.

3.6 Digital Solutions

During the COVID-19 pandemic, the large volumes of AE reports received by industry and regulators surpassed the projected numbers. Much was done (pre- and intra-pandemic) to augment the reporting platforms (from mobile applications to digital AE reporting forms and digital self-reporting surveillance systems) [5,6,7]. These digital tools were critical in allowing timely reporting and further supplemented active surveillance of post-vaccination symptoms [5]. Beyond these digital tools, there is an equal if not more important need to enhance the global data management capabilities and implement digital innovations that can further integrate reporting tools into healthcare systems, thereby allowing active surveillance through primary and secondary data analysis.

The digital solutions workshop discussed the importance of ensuring a robust digital infrastructure that supports the global delivery of key strategic priorities for pharmacovigilance and preparedness activities relating to surge capacity to handle increases in quantity or changes in quality of safety reports. Several challenges in implementing shared digital solutions were discussed in the workshop, including the following.

  • The lack of common data structure standards slows the time to analysis and interpretation of pharmacovigilance data sets.

  • The lack of compatibility of digital solutions with LMICs needs and lack of available alternatives. To address these challenges, a multi-stakeholder focus on data collection and data management is required. The following key strategic priorities were identified.

  • Modernizing and optimizing methods, including analytics, comparable insights (i.e., harmonization), and reporting of AEs. Digital transformation supports new approaches to safety monitoring and introduces intelligent automation from data collection to reporting. As both active and passive surveillance data collection becomes more automated (1) at the point of care; (2) via patient-reported outcomes, e.g. cohort event monitoring; (3) by access to large insurance data sets, methodologies and technologies such as generative artificial intelligence, machine learning models, rapid cycle analysis, real-time flow analytics, and powerful business intelligence platforms that support analytics, data quality assurance, and reporting services are bound to play an increasingly important role. Data integrity services, already critical to secure data against both intentional and unintentional data harm, are set to become increasingly important to support data quality and valid safety estimates.

  • Accelerated access to quality data by continuing to enhance capabilities related to active surveillance through primary and secondary data analysis. As digital transformation advances across LMICs, middle-income countries, and high-income countries, there will be increasing access to cost-effective large data sets. The quality of these emerging large data sets, from the perspective of pharmacovigilance, will be improved by ensuring there are guidelines specifying the data elements for each type of data set (e.g., insurance, hospital or electronic medical records, cohort event monitoring, etc.) and recommendations for case definitions/phenotypes best formulated for the specific data types.

  • Harmonization of both data and AE case definitions/phenotypes across program objectives, with creation and expansion of interoperable active surveillance networks to create large denominators that can support identification of rare AEs. The harmonization of data requires recognizing that different study or monitoring methods yield different types of data and that for each type of data there needs to be standardization of associated data structures to render them interoperable (e.g., insurance data, hospital or electronic health record data, cohort event monitoring, etc.). Simultaneously, there needs to be the possibility to apply the case definitions/phenotypes that are purpose built for each type of data, data structure, and associated data dictionary to support AE estimates and support meta-analysis or aggregated data analysis.

3.7 Low- and Middle-Income Countries

During the COVID-19 pandemic, mass vaccination campaigns took place in LMICs with the rapid introduction of novel vaccines. Although passive pharmacovigilance systems are in place in some LMICs, the advancement of healthcare professional knowledge about vaccine safety, IT infrastructure for pharmacovigilance data and resources, and human and financial resources to sustain these systems are required. There are various key players in addition to national pharmacovigilance centers, public health institutes/agencies, and regulatory agencies for vaccine safety surveillance in LMICs, including vaccine manufacturers, donors, non-governmental organizations; coordination of activities at the country level is essential. To address these challenges, the following top three key strategic priorities were identified.

  • Pharmacovigilance development or advancement/continuous education/knowledge sharing in the context of benefit–risk assessments: education of healthcare providers regarding the need for quality AE reporting was deemed a keen area of focus and need for LMICs at this time. Collaborative efforts between manufacturer-led educational programs and other non-governmental organizations was discussed. In addition, education of LMIC in-country regulatory staff on pharmacovigilance and benefit–risk analysis and providing associated peer-mentoring programs was also highlighted. Existing curricula and mentorship approaches, as well as the need to update them, were another focus of the workshop.

  • Resourcing, both human and financial: LMICs have limited resources for pharmacovigilance surveillance, including trained and dedicated staff and IT infrastructure. Although pilot programs are under way in some countries, the sustainability of existing programs is key in several countries (e.g., African Union Smart Surveillance [AU3S] funded by The Bill & Melinda Gates Foundation).

  • Coordination of activities at the country level: participants highlighted the lack of coordination/collaboration in some countries between Ministries of Health, National Regulatory Agencies, Epidemiology/National Immunization Programs, subnational agencies, and healthcare professionals. Sometimes, lack of coordination of activities is also linked to limited data access. There is a greater need for collaboration between these stakeholders.

In addition to these key priorities, access to the data sources for signal detection and vaccine benefit–risk assessment were also discussed. Vaccine adverse event following immunization databases in some countries may be under Ministries of Health/public health, so regulatory agencies may not have direct access to these data. Another key priority in LMICs for the monitoring of the introduction of a new vaccine is active vaccine safety surveillance to generate safety evidence more quickly and efficiently.

3.8 Transversal Session: Multi-Stakeholder Collaborations

The vaccine ecosystem is complex, with multifactorial challenges and interconnection of stakeholders, in terms of expertise, resources, and mandate, especially in the post-marketing setting. The COVID-19 pandemic pushed for more worldwide coordination and collaboration among different stakeholders and emphasized the need for resilient and ready-to-respond health structures/systems, for example, including resources to serve as “resting” capacity that can respond quickly with needed data. Although the spirit of collaboration may be lower since the pandemic, it is important to sustain the momentum. There are several ways to establish multi-stakeholder collaborations, either through worldwide, regional, or country-specific initiatives. Monitoring of vaccines requires data sources and expertise among many stakeholders. Given competing health priorities, cross-stakeholder collaboration should best serve public health. Several common challenges were highlighted during the conference.

  • Not one governance model fits all. Consequently, multi-stakeholder collaborations are often made at the project-initiative level, based on specific guiding principles and boundaries. Public and private collaborations are sometimes regarded with hesitancy, and questions have been asked whether shared goals and appropriate governance can transcend conflicting interests and mandates between public and private stakeholders, and whether benefits can outweigh the costs.

  • In parallel, since the COVID-19 pandemic, there is an increased public interest in disease surveillance and vaccine performance while rampant misinformation drives vaccine hesitancy. The recent push in real-world evidence, including digital expansion (multiple data sources) and participatory surveillance, has changed the role of patients in generating real-world data (e.g., smartwatch, home-swabs) within an evolving vaccine safety monitoring ecosystem. A higher level of transparency is required for disease surveillance and vaccine performance indicators to inform the public.

The following possible solutions and opportunities for multi-stakeholder actions were discussed.

  • Enable more frequent and broader multi-stakeholder cooperation and common projects offering higher visibility of common values, interests, and best practices guidance for cooperation in a range of options.

  • Promote open-source tools, open data access, and data-sharing practices in compliance with data protection laws to facilitate reproducibility and secondary use of data, together with appropriate governance for multi-stakeholder access.

  • Increase the opportunities for stakeholders to build trust through sharing both learnings and success stories (e.g. The Global Influenza Hospital Surveillance Network, Development of Robust and Innovative Vaccine Effectiveness [DRIVE] and COVIDRIVE).

4 Call to Action

With the success of the conference in bringing together stakeholders and in agreeing, in principle, on a set of priorities to address via collaboration, the next question is how best to continue this momentum to sustain the progress initiated by this forum while expanding the working alliance across all the stakeholders.

Although individual experts from multiple stakeholders are advancing some topics among working groups and proposing short-term actions following the June 2023 conference, post-event discussions between ‘onboarded’ stakeholders held in parallel have progressed toward identifying the key components that will define the future governance of BeCOME. There is no supra-national entity covering the BeCOME mandate, so a flexible governance framework is envisioned to host this scientific forum, facilitate stakeholders' connection, test innovative study designs, and catalyze new projects within the BeCOME areas of focus. Several governance guiding principles and public–private experiences were used to frame the future of BeCOME. The following framework was proposed as a starting point to engage discussions with potential hosting entities [8,9,10,11].

  • The governance should be as simple as possible, transparent, acceptable to all stakeholders, and appropriately sized to ensure efficiency. BeCOME should act as a matchmaker and catalyzer to foster synergies between various stakeholders and limit competitions and gaps. A project that may emerge from the BeCOME initiative should be managed as a stand-alone project with the appropriate stakeholders, based on its own governance and funding mechanisms. BeCOME should be considered as a facilitator for project initiation, a scientific forum for exchanges throughout a project’s lifespan, and an audience large enough to help disseminate outputs.

  • The roles, responsibilities, and decision-making rules should be agreed upon between the stakeholders willing to engage and be included in an agreement (at minimum, a memorandum of understanding). The level of engagement may differ between stakeholders depending on their interest and capacity. In-kind contributions and/or funds should be expected from institutions and companies. The structure and processes of the governance should consider mutual respect and shared benefits. The decision-making process should reflect a fair balance of these perspectives, especially between public and private stakeholders.

  • BeCOME should share its vision, objectives, guiding principles, contributing stakeholders, and planned activities on a public website. The dissemination of BeCOME outputs should target the scientific community at large, possibly through the organization of public online webinars, disclosure of proceedings from conferences, and publication of scientific manuscripts.

5 Next Steps

With the above considerations, the next immediate step post-conference was to sustain the current scientific exchange forum, advancing discussions on methods and guidance with multi-stakeholder representatives and expert groups while welcoming scientists and experts in the field to promote field testing and evaluation of novel approaches to the study of the risks and benefits of vaccination. The subsequent step involves forming a public–private partnership model(s), which will require some time to establish formal agreements but can potentially deliver funding to sustain key priorities for the future. There may be several of those formal public–private partnerships under different expert groups, but all established under the overarching umbrella of BeCOME.

6 Conclusions

The conference was successful in creating an environment for open dialogue among different stakeholders. Common challenges and priorities were agreed upon, along with the need for robust, mutually beneficial solutions. Additionally, within each of the six scientific topics (BIR, safety studies, pregnancy surveillance, vaccine benefits, signal detection and digital solutions), the specific needs of LMICs were evaluated. The multi-stakeholder collaboration session highlighted proposals for formalizing continued collaboration, expanding to a broader set of public and private stakeholders, and thus to advance the BeCOME mission to “Foster collaborations where experts in pharmacovigilance and pharmacoepidemiology from key stakeholders (public health, regulators, academics, industry) can openly discuss and collectively align to sustain relevant systems, improve processes, and develop innovations for the post-marketing monitoring of benefits and risks of vaccines, leveraging the learnings and keeping the momentum of the COVID-19 emergency.”