Pivoting data and analytic capacity to support Ontario’s COVID-19 response

Abstract

Introduction
Health care systems have faced unprecedented challenges due to the COVID-19 pandemic. Access to timely population-based data has been vital to informing public health policy and practice.

Methods
We describe how ICES, an independent not-for-profit research and analytic institute in Ontario, Canada, pivoted existing research infrastructure and engaged health system stakeholders to provide near real-time population-based data and analytics to support Ontario's COVID-19 pandemic response.

Results
Since April 2020, ICES provided the Ontario COVID-19 Provincial Command Table and public health partners with regular and ad hoc reports on SARS-CoV-2 testing and COVID-19 vaccine coverage. These reports: 1) helped identify congregate care/shared living settings that needed testing and prevention efforts early in the pandemic; 2) provided early indications of inequities in testing and infection in marginalized neighbourhoods, including areas with higher proportions of immigrants and visible minorities; 3) identified areas with high test positivity, which helped Public Health Units target and evaluate prevention efforts; and 4) contributed to altering the province's COVID-19 vaccine roll-out strategy to target high-risk neighbourhoods and helping Public Health Units and community organizations plan local vaccination programs. In addition, ICES is a key component of the Ontario Health Data Platform, which provides scientists with data access to conduct COVID-19 research and analyses.

Discussion and Conclusion
ICES was well-positioned to provide rapid analyses for decision-makers to respond to the evolving public health emergency, and continues to contribute to Ontario's pandemic response by providing timely, relevant reports to health system stakeholders and facilitating data access for externally-funded COVID-19 research.

Introduction

In late January 2020, Ontario (Canada’s most populous province) reported its first case of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, or coronavirus disease (COVID-19). Like all jurisdictions around the world, Ontario’s health system faced unprecedented and evolving challenges due to the global pandemic. The need for timely responses to address rapidly emerging questions was crucial, with much being unknown about which groups were at higher risk and what the most effective measures were to reduce the risk of contracting the virus and disease.

Data and analytics have been crucial to inform and plan strategies to mitigate COVID-19’s impact to the health system in Ontario. Early on in the pandemic, historical data on hospital and intensive care unit admissions were used to estimate hospitals’ surge capacities [1], and to model the impact of public health interventions against best- and worst-case scenarios for health care resources [2]. Near real-time data and analyses were used to describe the evolution of COVID-19 at the population-level and in high-risk populations to plan targeted preventive efforts and inform policy. Data collected on confirmed COVID-19 cases and COVID-19-related interventions such as hospitalization, intensive care and mechanical ventilations revealed groups at high risk of both COVID-19 infection and severe disease, which informed vaccination strategies [3].

The unprecedented need for timely and relevant analyses to support Ontario’s health system revealed some limitations in the province’s existing health data analytics capacity. Many of the datasets required were not initially coordinated (e.g., SARS-CoV-2 testing data was de-centralized) or not readily available for real-time analyses (e.g., data were compiled and sent monthly for physician and pharmacy billings and vital statistics updates, and quarterly for hospitalizations and emergency department visits). Furthermore, multiple organizations had the capability to perform these analyses and coordination was needed to maximize efficiency, ensure consistency, and avoid duplication. Prior to the World Health Organization’s declaration of the COVID-19 pandemic on 11^th March, 2020, ICES (formerly the Institute for Clinical Evaluative Sciences) - an independent not-for-profit research and analytic institute that houses administrative health and demographic data for all Ontario residents [4] – initiated efforts to receive real-time SARS-CoV-2 testing data for the province and accelerate feeds for other data sources, with the goal of linking these data with other population-level datasets to enable a range of research projects and data analytics on COVID-19. With its existing expertise and infrastructure, its mission to inform policy and practice through relevant and timely work, and familiarity working with scientists, researchers and policymakers under pressing timelines, ICES was well-positioned to support decision-makers in the rapidly evolving public health emergency. Furthermore, ICES’ status under Ontario law as a prescribed entity that can receive and use health information without consent for the purposes of compiling and analyzing statistical information about the health system in Ontario enabled expedited access to data and analytics [5].

Here we summarize how ICES adapted to address some of Ontario’s pandemic data needs. In particular, we describe one project that provided the Ontario COVID-19 Provincial Command Table with evidence to inform public health policy and practice using detailed information from our repository of routinely collected, population-based administrative health data, linked to SARS-CoV-2 testing and COVID-19 vaccination data. The initial goal was to support the development of a provincial testing strategy by describing sociodemographic and clinical characteristics of Ontarians who received SARS-CoV-2 diagnostic testing, identifying populations who were not being tested and ascertaining groups most at risk of testing positive for SARS-CoV-2. As the pandemic advanced, the needs of our Knowledge Users changed and we currently include analytics on COVID-19 vaccinations.

Methods

Setting

ICES receives funding from Ontario’s Ministry of Health to respond to analytic requests from health system stakeholders through the Applied Health Research Question (AHRQ) Program. An AHRQ is a question asked by a “Knowledge User” to generate evidence to inform planning, policy and program development [6, 7]. Eligible Knowledge Users include government ministries, hospitals, Public Health Units (PHUs), health care providers and Indigenous organizations and communities [6, 8]. In response to the pandemic the Ministry of Health launched the Ontario Health Data Platform in early 2020 to support COVID-19 research, which included ICES as a key partner and financially supported the expansion of the scope of the ICES AHRQ Program for rapid COVID-19-related requests.

Within a month of the pandemic being declared, an AHRQ was submitted to provide the Ontario COVID-19 Provincial Command Table (comprised of representatives from multiple health agencies and ministries) with information on who was being tested and positive for SARS-CoV-2. There was an urgency to understand the sociodemographic and clinical characteristics of these individuals to strategize prevention efforts and determine how testing and test positivity progressed within vulnerable populations such as residents of various congregate care/shared living settings (e.g., long-term care facilities, retirement homes, group homes, shelters and correctional facilities) [9]. The questions and needs of the Knowledge Users evolved as more was understood about the virus’ distribution geographically and within various risk groups, the pandemic’s disproportionate impact on immigrants/refugees and marginalized communities, and as strategies for vaccine distribution were being developed.

Data sources

Early in the pandemic, several challenging data gaps were evident. To facilitate rapid turnaround of analyses, ICES worked with new and existing data providers to receive more frequent feeds of routinely available datasets and to expand our data assets to include new data critical to COVID-19 reporting. This required expedited negotiation of data partnerships and privacy and legal reviews. We also adjusted internal workflows to process these datasets more rapidly (e.g., data cleaning, standardization, quality assessments) upon receipt to make them promptly available for analyses. Below we summarize the data sources, how they were used for COVID-19 analytics, any accommodations to ensure timely reporting, and limitations. A detailed description of some of these data sources is described elsewhere [4].

Population and demographics

The Ontario Health Insurance Plan (OHIP) Registered Persons Database (RPDB, which has demographic information for individuals covered by OHIP) was used to determine the postal code of residence for individuals to enable reporting at smaller geographical areas and by neighbourhood-level characteristics (e.g., income quintile and marginalization indices, which include dimensions such as residential instability, material deprivation, dependency [proportion of the population without employment income], and ethnic diversity) [10]. RPDB also was used to estimate the number of individuals living in each geographical area to calculate per capita SARS-CoV-2 testing rates and COVID-19 vaccine coverage. Prior to the pandemic, ICES processed RPDB every two months, but this changed to monthly to enable allocation of individuals to the most current geographical units as of testing/vaccination date. Although RPDB has high levels of completeness for basic demographic information [11], the currency of address and vital status depend upon notifications to the Ministry of Health. For example, updates to address information are relayed during health care encounters and health card renewals, but the latter were suspended during the pandemic [12]. Death information can be delayed for even more than a year and relies on family or health practitioner of the deceased communicating this information. However, ICES enriches RPDB with more up-to-date death information through linkage to other administrative databases [13].

The Ontario portion of Immigration, Refugees, and Citizenship Canada’s Permanent Resident database was used to characterize individuals by immigration status (e.g., immigrated as refugees or through other permanent resident classes [i.e., sponsored family, economic class]) [14]. However, our current Immigration, Refugees, and Citizenship Canada data holding only includes individuals who landed as permanent residents in Ontario between 1^st January 1985 and 31^st May 2017. Those who registered more recently with OHIP were classified as newcomers to Ontario, although they may not all be immigrants.

Health services

The OHIP claims database contains claims for physician services covered by the province’s insurance plan and its mandatory data elements are highly complete as they are required for payment purposes [11]. However, some laboratory tests, selected procedures provided to inpatients, and services by physicians of alternative funding plans that do not shadow-bill are missing from this database. The Ontario Drug Benefit (ODB) database contains prescription drug claims made to the province’s drug insurance program (which is available to individuals aged ≥65 years and those receiving social assistance). Like the OHIP database, the ODB database is highly complete for mandatory data elements, as these are required for pharmacy payment [15]. These datasets, along with the long-term care Continuing Care Reporting System database, were used to identify residents of long-term care homes. Since long-term care residents were a priority group to monitor and because the Continuing Care Reporting System database was only updated quarterly, the OHIP and Ontario Drug Benefits databases were processed monthly instead of every two months.

Other data holdings such as the Canadian Institute for Health Information Discharge Abstract Database (inpatient hospitalizations) and the National Ambulatory Care Reporting System database (emergency department visits) were used to identify individuals with comorbidities [16]. In addition, the Discharge Abstract Database was used to identify mother-baby dyads from birth hospitalizations and family units. These databases capture all visits to hospitals and emergency departments in Ontario and the completeness of mandatory data elements is required for hospital payment purposes. However, although hospitals are required to submit records of visits within a specified timeframe, timeliness of submission varies widely by institution. These databases were normally updated every quarter but were instead processed monthly.

COVID-19-specific datasets

Initially, only a few laboratories in Ontario were performing SARS-CoV-2 testing and test results were not stored in a centralized repository. Eventually, testing became more widespread and results were centralized, and in April 2020, ICES started receiving a daily feed of SARS-CoV-2 real-time polymerase chain reaction test results contained in the Ontario Laboratories Information System (OLIS). Since 2016, ICES has received quarterly feeds of OLIS covering the majority of Ontario’s laboratory tests results and had developed expertise in working with its complex data structure. However, the sheer volume and velocity of the daily cumulative tests and the text-based format of the SARS-CoV-2 test results presented new challenges. Before the pandemic, work to interpret respiratory virus test results in OLIS was underway, and this methodology was adapted for the SARS-CoV-2 tests. ICES collaborated closely with teams at the Ministry of Health, Public Health Ontario, and clinical subject matter experts to interpret, validate, and develop an algorithm to transform these data into a research-friendly format. The code to parse relevant test results from these data was published under an open-source license [17] and was subsequently used by the Ministry of Health and Ontario Health for their COVID-19 analytics. Adoption of this code by other organizations that were receiving the same OLIS data feed enabled consistent interpretation of SARS-CoV-2 test results and facilitated uniform reporting of COVID-19 metrics.

Congregate care/shared living settings emerged as high-risk environments for SARS-CoV-2 outbreaks and identifying residents of these settings was challenging using existing health administrative databases. Currently, only long-term care residents can be identified using records from health administrative databases. However, for other congregate care/shared living settings, different methods were developed. In particular, OLIS contains the addresses of the facilities where the tests were administered and these addresses were matched to master lists of addresses for retirement homes, correctional facilities, and shelters using fuzzy address matching techniques to identify individuals who were likely residing in these settings at the time of SARS-CoV-2 testing [18].

Information on COVID-19 cases such as symptoms, epidemiological contacts and risk factors are not complete in the OLIS data nor are captured in other health administrative databases. Thus, to enable more descriptive reporting on COVID-19 cases, ICES started receiving daily feeds of the Public Health Case and Contact Management (CCM) Solutions database, the centralized database containing all confirmed COVID-19 cases in Ontario, which was an entirely new data holding for ICES. CCM was used in combination with OLIS to identify all individuals confirmed positive for SARS-CoV-2 because there were differences in capture rates between the two data sources: OLIS captures approximately ~90% of all confirmed cases that are reported in CCM, but approximately ~6% of cases in CCM are not linkable to other ICES data holdings.

In March 2021, ICES started receiving weekly cumulative data feeds of COVID-19 vaccine administration events recorded in the Ontario COVID-19 vaccination registry (COVax_ON. For both CCM and COVax_ON, ICES performed deterministic and probabilistic record linkage using identifiers other than the provincial health insurance number (e.g, full name, date of birth) to identify additional individuals who could be linked to other data sources.

Analysis and reporting

Numerous questions and ideas were proposed by Knowledge Users and ICES Scientists. However, competing priorities with existing research and analytic projects and limited human resource capacity posed challenges to addressing them all. Accordingly, ICES created a committee of staff and scientists to meet twice weekly to help coordinate COVID-19-related projects and data infrastructure (e.g., data acquisition and prioritization). This group engaged with policymakers and Knowledge Users to scope how ICES could best support the pandemic response. Several members of the ICES COVID-19 Committee also had roles on provincial advisory groups (AG, JCK, LCR, MJS) and had direct exposure to ongoing discussions on pressing policy topics and were able to assess where ICES data and analytics could help address knowledge gaps. In addition, members had familiarity with key data holdings (e.g., long-term care datasets), experience with other datasets that could be applied to COVID-19 research (e.g., laboratory and vaccination data) and expertise in public and population health concepts. Collaboratively, this group devised research and analytic plans to understand patterns and limitations seen in the data, interpreted results, and provided recommendations to enhance utility of the reports. The Committee also worked closely with PHUs to customize reports most valuable to their jurisdictions. As the pandemic evolved, all Knowledge Users were continuously engaged to refine analytic approaches, develop new metrics and reports, and determine optimal frequency of reports. The utility of report contents was frequently re-evaluated with partners and revised to ensure relevance and timeliness. By-products of some of these reports were shared publicly on the ICES COVID-19 Dashboard, including downloadable data tables, and updated weekly [19].

Results

Here we highlight the frequent COVID-19 analytics and reports (and selected ad hoc reports) provided by ICES starting in April 2020 and describe their impact on practice and policy.

1. Daily congregate care/shared living testing reports

Based on early requests from health system stakeholders, testing and positivity rates in long-term care homes and retirement homes were an initial area of focus. ICES was the first organization to report on SARS-CoV-2 testing rates and laboratory-confirmed COVID-19 cases for each long-term care facility and retirement home in Ontario, though we could not report on testing rates in retirement homes because there is currently no method to identify residents using administrative data. Providing near real-time reports early in the pandemic helped identify facilities for targeted testing and prevention efforts.

2. Characteristics of individuals tested and positive for SARS-CoV-2

Another goal was identifying additional high-risk populations. ICES characterized all Ontarians eligible for SARS-CoV-2 testing and identified groups (e.g., sociodemographic, clinical) that were disproportionately affected by low rates of testing yet had high positivity. These analyses provided a snapshot of who was being tested and testing positive during the prior two weeks, which was determined a more appropriate timeframe than the cumulative period to reflect the frequently changing testing strategy and criteria. Reports on children and youth aged <19 years were also generated to inform discussions regarding school re-opening.

On 11^th May 2020, the first public-facing report was released [20], suggesting there was a disproportionate impact of COVID-19 on individuals in certain neighbourhoods. For example, there was a higher proportion of individuals residing in marginalized neighbourhoods among individuals tested for SARS-CoV-2 compared to those not tested (e.g., greater residential instability 27% vs. 23%; material deprivation 21% vs. 18%, dependency 24% vs. 17%, and lower income 23% vs. 20%, Figure 1). Conversely a lower proportion of those tested lived in areas with high ethnic diversity (i.e., higher numbers of immigrants and visible minorities, defined by Statistics Canada as “persons, other than Aboriginal peoples, who are non-Caucasian in race or non-white colour”) [21]. This group also comprised a large proportion (41%) of confirmed COVID-19 cases.

Figure 1: Proportion of individuals in Ontario not tested, tested, and confirmed positive for SARS-CoV-2 living in the most marginalized neighbourhoods by neighbourhood sociodemographic characteristic* (Exhibit from first public-facing report† on the characteristics of individuals tested for SARS-CoV-2 in Ontario, Canada as of 30th April 2020). ^*Percentages represent the proportion of all people in each ‘Not Tested’, ‘Tested’ or ‘Confirmed Positive’ group that reside in the most marginalized group for the neighbourhood characteristic of interest. Compared to individuals not tested for SARS-CoV-2, a higher proportion of those tested reside in marginalized neighbourhoods (e.g., greater residential instability, material deprivation and dependency, lower income). Also, individuals confirmed positive for SARS-CoV-2 were more likely to live in neighbourhoods with higher proportions of immigrants and visible minorities. Marginalization was measured using the dimensions of the Ontario Marginalization Index. ^†From Chung et al. ‘COVID-19 Laboratory Testing in Ontario: Patterns of Testing and Characteristics of Individuals Tested, as of April 30, 2020’. Toronto, ON: ICES; 2020

A subsequent in-depth report focusing on immigrants, refugees, and newcomers to Ontario corroborated emerging evidence of the inequities of COVID-19 in Ontario [22]. In response to this report, 3 PHUs with large immigrant and refugee populations requested analyses focusing on their residents. These results were shared with community organizations and used to improve access to testing.

3. Testing rates and percent positivity by smaller areas of geography

Early in the pandemic, it became apparent that reporting at levels of geography smaller than a PHU would be helpful to plan targeted prevention efforts (e.g., pop-up testing clinics, directing surveillance efforts) and monitor progress. Four PHUs requested these measures at geographical units that aligned with areas used for their health promotion and community outreach efforts.

ICES began generating these measures at the level of forward sortation areas (FSA) – the first 3 digits of the Canadian postal code – for which the population size ranges from 500 to 125,000 residents. Testing metrics by FSA were shared with PHUs on a weekly basis.

On the ICES COVID-19 Dashboard, we plotted percent positivity by testing rate per capita for each FSA within selected PHUs to show the variability of COVID-19 within a PHU and to highlight areas in need of targeted interventions (Figure 2).

Figure 2: Example exhibits on SARS-CoV-2 testing from ICES COVID-19 Dashboard‡ - Per capita testing and percent positivity for COVID-19 by Forward Sortation Areas (FSA) in selected Public Health Units in Ontario for the week of 8th to 14th November 2020§. ^‡From ICES COVID-19 Dashboard (https://www.ices.on.ca/DAS/AHRQ/COVID-19-Dashboard; posted on 19^th November, 2020). ^§Percent positivity is plotted against testing rate per capita for each FSA, which is the first three characters of a postal code, to show the variability of COVID-19 within a Public Health Unit. Size of each circle is proportional to the number of cases per 100,000.

4. COVID-19 vaccine coverage

At the request of PHUs, in February 2021 ICES provided estimates of the numbers of individuals living in small geographic areas stratified by age group, which were used for vaccine distribution planning.

On 30^th March, 2021, we compiled vaccine coverage estimates by age group based on neighbourhood COVID-19 risk to respond to an urgent request from the Ontario COVID-19 Science Advisory Table [23] to determine whether COVID-19 vaccine distribution was aligned with those at highest risk [24]. Initial results (using vaccination events up to 22^th March, 2021) demonstrated large disparities in vaccine coverage between neighbourhoods/FSAs stratified based on cumulative COVID-19 incidence, with vaccine coverage substantially lower in high-risk areas (5% vs. 9% among all eligible age groups combined [≥16 years at that time]), and 38% vs 49% among those aged ≥80 years). An updated analysis released on the ICES COVID-19 Dashboard a week later garnered significant media attention and contributed to rapid changes in the vaccine strategy. On 6^th April 2021 the Ontario government announced plans to target neighbourhoods with the highest risk of infections (“hot spots”). Since then, ICES has updated these analyses weekly to monitor the vaccine roll-out and assess the impact of changes in the vaccine strategy (e.g., opening pop-up clinics and lowering the age criteria in “hot spots”, gradually lowering age criteria) and shared these results with Ontario’s COVID-19 Vaccine Distribution Task Force and Science Advisory Table (Figure 3). We also compiled this analysis according to immigrant status, which showed lower vaccine coverage compared to long-term residents/Canadian-born individuals, for the federal Privy Council Office advising the Prime Minister and the federal immigration ministry, Immigration, Refugees and Citizenship Canada.

Figure 3: Example exhibits on COVID-19 vaccine coverage from ICES COVID-19 Dashboard** - Percentage of Ontarians who have received at least 1 dose of a COVID-19 vaccine (vaccine coverage) by age group and neighbourhood COVID-19 infection risk as of a) 29th March 2021 and b) 19th April 2021#. **From ICES COVID-19 Dashboard (https://www.ices.on.ca/DAS/AHRQ/COVID-19-Dashboard; posted on 1^st April 2021 [a] and 22^nd April 2021 [b]). ^#FSAs were grouped into deciles (each representing a neighbourhood risk group) based on the cumulative incidence of COVID-19 infection among residents living outside of long-term care facilities within each FSA as of 28^th March, 2021. ^††Vaccine coverage is the number of residents in each age group and neighbourhood risk stratum who received at least 1 dose of a COVID-19 vaccine, divided by the total number of individuals in the stratum (excluding long-term care residents).

We also provided PHUs with cumulative estimates of individuals who received at least one COVID-19 vaccine dose, overall and by age groups weekly, which they used to monitor vaccine uptake and focus their efforts accordingly.

Discussion

With extensive staff expertise and established authorities and mechanisms, ICES rapidly pivoted to support decision-makers’ response to an emerging public health crisis, and provided researchers with access to large health datasets through the Ontario Health Data Platform [25] for COVID-19-related research. Complementary scientific, public health, and clinical expertise, existing and new partnerships with key data providers, proactive engagement with health system stakeholders (many of whom ICES collaborated with previously), expertise in data linkage and quality, and established legal and privacy processes enabled ICES to provide relevant and timely data and analytics critical to public health policy and practice. Although ICES regularly responds to Knowledge Users’ requests for custom analytics, we primarily conduct scientist-driven studies which involve more flexible timelines to analyze, interpret, and communicate results. Due to the pandemic, there was a need to adapt our evidence generation processes, which included the need to shelve non-COVID-19-related studies temporarily to enable weekly reporting to health system stakeholders and the public.

While ICES had well-established relationships with policymakers prior to the pandemic, new relationships with Knowledge Users were established to ensure that reports were relevant. For examples, many PHUs were previously unfamiliar with ICES, its data protection safeguards, and its access to individual-level provincial health administrative data. This required frequent communication and continuing education to help these teams interpret the analytics and how the information could inform their work. In addition, as data and analytic capacity grew among other organizations, ICES took a proactive approach to avoid duplication by sharing methods and outputs, and engaging with teams to understand differences in methodologies, interpretation, and limitations.

The urgency for timely data and analytics presented unprecedented challenges, which required ICES departments to adapt some of their normal operations. Lessons learned from these challenges would benefit future big data and research endeavours (Table 1). The acquisition of novel datasets and frequent analytics and reporting required several departments to devote human resources for COVID-19-related tasks, which consequently diverted them away from normal research operations. As the pandemic continued, ICES frequently reassessed the analytic and reporting needs of the Knowledge Users to determine whether human resources could be reallocated to pre-pandemic planned projects so that those projects could resume.

Table 1. Challenges, adaptations, and lessons learned in response to the data needs of Ontario’s COVID-19 pandemic response by ICES department.

Department	Examples of Challenges	Adaptations	Lessons Learned
All departments	Public sector research institute with limited staff resources, who were all adjusting to working from home and managing other responsibilities.	Reallocated human resources from low priority, non-COVID-19-related work to COVID-19-related tasks. Worked long hours and on weekends to accommodate timely data and analytic requests	Engaged, dedicated, and committed human resources are imperative to organization’s success
Communications	Weekly analyses and data updates posted to ICES website via COVID-19 Dashboard, often with novel and actionable findings, garnered significant number of media requests and social media engagement	Prioritized media requests and accommodated availability of ICES scientists Social media conversations responded to within 24 hours	Need for active engagement in discussions with ICES scientists on interpretation and messaging of results, and collaboration with research teams to present lay language summaries and other data visualizations Response to social media is vital to engagement with the public, but not all comments needed to be addressed
Data Quality & Information Management	Frequent (initially daily and cumulative) data feeds of new SARS-CoV-2 test result data in text-based format contained rich information but data format was novel and not research-ready Existing clinical and administrative data feeds, which were historically updated bimonthly or quarterly, were not ideal for real-time analyses	Prioritized resources to accommodate data processing (e.g., linkage, standardization, data quality assessments) for frequent data feeds Leveraged new, innovative methods (e.g, text-mining, address matching) to enhance use of COVID-19 related data holdings Expedited posting data (new and historical feeds) in analytical environment for research teams Shared developed tools under open access license for external stakeholders to achieve consistent data interpretations and to circumvent the need to disclose cleaned data to partner organizations	Invest in skill building in other programming language and data cleaning methods to prepare novel datasets and data needs Engage with data providers early to establish processes for data transfers (and contingency plans for delayed transfers) and notification for changes in data structure or contents Utilizing previously established relationships with data providers and content experts allows for immediate cooperation when problem-solving data issues
Information Technology	Existing data analytic environment and infrastructure were not optimal for data processing, computing, and storage of large volumes of data frequently. File format for SARS-CoV-2 testing data (SAS datasets) not efficient for tools used to analyze data (e.g., Python use to clean text-based results)	Implemented enhancements to analytic environments, including the high-performance computing cluster, to expand data storage capacity and improve performance Modified scheduled maintenance to accommodate COVID-19 reporting schedule	Invest in data processing tools (e.g., automation) and skill sets to reduce dependency of manual processes Understanding of the research needs and timelines (e.g., volume of new data and frequency of reporting) by having a more efficient work intake process would help assess data infrastructure needs (e.g., storage, analytic tools)
Privacy and Legal Office and Cybersecurity	Large number of COVID-19-related research and/or analytic projects proposed that aimed to address timely and relevant questions at the same time as other non-COVID-19-related projects requiring privacy support Large number of COVID-19-related requests for data acquisition and data sharing Process to acquire and collect new data requires diligent review before data transfer and utilization	Privacy team enabled prioritization of privacy impact assessments for COVID-19-related projects, data acquisition and data sharing agreements Division of labour amongst expanded team of privacy impact assessment reviewers enabled prioritization of certain requests Policies, agreements, and safeguards were revised to allow scientists and staff to work with ICES data securely while remote	More staff trained and focused on privacy, enables greater capacity to take on work, support shorter timelines and prioritization, while maintaining diligence, integrity, and trust Ongoing and frequent communication with the data partner disclosing data to ICES allowed for understanding of legalities earlier in the data acquisition process
Public Engagement & Knowledge Translation	Need for rapid, public-friendly and accessible materials on COVID-19 data, analytics, and information for public consumption	Prioritized public engagement activities and designed plain language online resources to increase accessibility, understanding, and awareness of COVID-19 related research and ICES COVID-19 Dashboard Convened Public Advisory Council subcommittees to provide rapid turn-around input on certain COVID-19 reports (e.g., in-depth report on SARS-CoV-2 testing among immigrants, refugees, and newcomers to Ontario) Shifted to virtual settings to conduct regular Public Advisory Council meetings and working groups	Input from the Public Advisory Council remains vital for ensuring that the information that ICES puts out on its web-based and social platforms is public-friendly and easily accessible Leveraging members of the Public Advisory Council for nimble input on time sensitive materials for which public input was critical to framing and interpretation of results
Research and Analysis/Data and Analytic Services	Urgency to use newly acquired COVID-19 datasets, which had unfamiliar data structures, contents, and limitations Some Knowledge Users were not previously acquainted to health administrative data and its value for decision making Frequent reporting and tight timelines conflicted with other research project commitments	Created analytical datasets, macros, & definitions to be used across COVID-19 reports, and shared among research teams Initiated working groups with data providers and other organizations to understand new datasets and share knowledge Prioritized resources to accommodate rapid increase in COVID-19 projects	Establishing data working groups within ICES and with other organizations (including teams which ICES has collaborated successfully in the past) to share knowledge about datasets, utility, and limitations is beneficial to ensure consistency concepts and interpretation, and to avoid duplication of efforts Building redundancy (i.e., multiple staff with same knowledge) on dataset contents, analyses, and key deliverables is imperative to projects with unpredictable timelines
Science Office	Competing priorities and timelines between existing investigator-led research projects and work that contributed to the pandemic response Multiple COVID-19 projects proposed by ICES Scientists, often with overlapping objectives	Established committee of staff and scientists to prioritize COVID-19 related initiatives and research projects. Communicated openly with ICES investigators regarding potential delays in pre-existing work Led discussions with Knowledge Users about analytical plans and reports, conceptualized data visualizations and interpreted results for ICES COVID-19 Dashboard. Accommodated media requests to discuss about results presented on the ICES COVID-19 Dashboard, often with short notice.	Prioritization of projects based on objectives and relevance is needed when human resource capacity is limited Fostering collaboration between project teams with similar research interests and complementary knowledge/skills strengthens quality of research impact
Strategic Partnerships	Urgent need to acquire new datasets (often with new data partners) with timely and relevant information to inform pandemic response	Negotiated with existing partners and established new partnerships expeditiously	Building and maintaining strong partnerships with all stakeholders and data custodians enables timely acquisition of data when urgently needed

Rapid analytics can be challenged by data availability and quality issues, which necessitate the application of expertise in a new domain. For example, since ICES did not have methods to identify residents of non-long-term care facilities, we leveraged data science expertise already in-house to embark on innovative data cleaning approaches enabling matching of facility addresses to non-standardized addresses recorded on laboratory requisitions. The utility of the retirement home testing reports validated these new approaches. Another example was for a request to assess a May 2020 testing “blitz” in long-term care facilities using unique outbreak investigation numbers. The challenge was two-fold: 1) these numbers were poorly captured in OLIS, and 2) where present, they were embedded in text-fields, which required our data science experts to develop new data cleaning techniques.

Urgent data requests also presented opportunities for creative applications of pre-existing ICES research. One example was in response to report on SARS-CoV-2 testing volumes in school-aged children and their family members to inform the testing criteria in September 2020. However, our current data holdings do not identify a family unit. Instead, we used the ICES-derived Ontario Mother-Baby Linked Dataset to identify siblings born to the same birth mother using the mother’s unique identifier. Another example was when we compiled estimates by vaccine priority groups for PHUs’ vaccine planning and were unable to comprehensively identify chronic home care recipients (a priority group in the vaccination roll-out’s initial phase [26]) because our current data holdings could only identify recipients as of September 2020, which was 3-6 months before the vaccine roll-out. These opportunities forced us to conduct an environmental scan across ICES research projects to seek alternative methods to overcome the data limitations.

Finally, one of ICES’ unique contributions to the pandemic response was the ICES COVID-19 Dashboard. There was an urgency to share information with the public so Ontarians could take individual actions to protect themselves and their communities. Working collaboratively with our Communications and Public Engagement and Knowledge Translation teams, and ICES’ Public Advisory Council, each week the ICES COVID-19 Committee rapidly interpreted results, developed data visualizations, and synthesized key findings. Although there are several organizations with dashboards, ICES focuses on presenting information based on its linked data holdings. ICES was the first to report testing and vaccination metrics at the FSA-level in Ontario, which was chosen because the public would readily be able to identify their area-level risk based on their own postal code. ICES also shared aggregate data on these metrics as an open data source. Many health journalists use these data to publish their own data visualizations and inform their stories. Provincial modelling tables, community organizations, and other independent data analysts also use these open data for their analyses, community outreach initiatives, and personal COVID-19 dashboards.

Conclusion

The COVID-19 pandemic highlighted the important need for near real-time data and analytics to help inform COVID-19 testing and vaccination policies, strategies, and practices. Although needing to operate under time and resource constraints, ICES was well-positioned to provide rapid analytics to Knowledge Users and health system stakeholders to inform policy and practice and facilitate data access for a broad range of researchers through open data and the Ontario Health Data Platform, all of which continues to contribute to Ontario’s ongoing pandemic response.

Ethics statement

ICES is a prescribed entity under section 45 of Ontario’s Personal Health Information Protection Act (PHIPA). Section 45 authorizes ICES to collect personal health information, without consent, for the purpose of analysis or compiling statistical information with respect to the management, evaluation, monitoring or planning for all or part of the health system. Applied Health Research Question (AHRQ) projects are typically conducted under section 45 of PHIPA and do not require review by a research ethics board.

Declaration of conflicts of interest (COI)

All authors are employees of ICES, which is funded by the Ontario Ministries of Health and Long-Term Care.

Acknowledgements

This work is supported by the Applied Health Research Questions (AHRQ) Portfolio at ICES, which is funded by the Ontario Ministry of Health. For more information on AHRQ and how to submit a request, please visit www.ices.on.ca/DAS/AHRQ. This work is also supported by the Ontario Health Data Platform (OHDP), a Province of Ontario initiative to support Ontario’s ongoing response to COVID-19 and its related impacts. Parts of this material are based on data and information compiled and provided by Ontario Ministry of Health, the Canadian Institute for Health Information, Public Health Ontario and Immigration, Refugees and Citizenship Canada (IRCC) current to May 2017. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of ICES, the OHDP, IRCC or the funding or data sources; no endorsement is intended or should be inferred. We would like to acknowledge Public Health Ontario for access to case level data from CCM and COVID-19 laboratory data, as well as assistance with data interpretation. We also thank the staff of Ontario’s public health units who are responsible for COVID-19 case and contact management and data collection within CCM.

References

1. Mishra S, Wang L, Ma H, Yiu KCY, Paterson JM, Kim E, et al. Estimated surge in hospital and intensive care admission because of the coronavirus disease 2019 pandemic in the Greater Toronto Area, Canada: a mathematical modelling study. CMAJ Open 2020;8:E593–604. 10.9778/cmajo.20200093.

   https://doi.org/10.9778/cmajo.20200093

2. Tuite AR, Fisman DN, Greer AL. Mathematical modelling of COVID-19 transmission and mitigation strategies in the population of Ontario, Canada. CMAJ 2020;192:E497–505. 10.1503/cmaj.200476.

   https://doi.org/10.1503/cmaj.200476

3. Papst I, Li M, Champredon D, Bolker BM, Dushoff J, Earn DJD. Age-dependence of healthcare interventions for COVID-19 in Ontario, Canada. BMC Public Health 2021;21. 10.1186/s12889-021-10611-4.

   https://doi.org/10.1186/s12889-021-10611-4

4. Schull MJ, Azimaee M, Marra M, Cartagena RG, Vermeulen MJ, Ho M, et al. ICES: Data, discovery, better health. Int J Popul Data Sci 2019;4. 10.23889/ijpds.v4i2.1135.

   https://doi.org/10.23889/ijpds.v4i2.1135

5. ICES - Privacy at ICES n.d. https://www.ices.on.ca/Data-and-Privacy/Privacy-at-ICES (accessed September 16, 2021).

6. Research at the Ministry of Health - Health Care Professionals - MOH n.d. https://www.health.gov.on.ca/en/pro/ministry/research/applied_heq.aspx (accessed May 31, 2021).

7. Ishiguro L, An D, Plumptre L, Paul J, Mecredy G, Li K, et al. Supporting policy and practice in Ontario through ICES’ Applied Health Research Question (AHRQ) Program. Int J Popul Data Sci 2021;6:20. 10.23889/IJPDS.V6I3.1683.

   https://doi.org/10.23889/IJPDS.V6I3.1683

8. ICES - Applied Health Research Question n.d. https://www.ices.on.ca/DAS/AHRQ (accessed May 31, 2021).

9. Public Health Ontario. COVID-19 Resources for Congregate Living Settings |Public Health Ontario. 2020 n.d. https://www.publichealthontario.ca/en/diseases-and-conditions/infectious-diseases/respiratory-diseases/novel-coronavirus/congregate-living-settings-resources (accessed September 16, 2021).

10. Toronto, ON; 2016.

11. Goel V, Williams J, Anderson G, Blackstien-Hirsch P, Fooks C, Naylor C. Patterns of Health Care in Ontario. The ICES Practice Atlas. 1996.

12. ServiceOntario: COVID-19 (Coronavirus) response | Ontario.ca. 2020 n.d. https://www.ontario.ca/page/serviceontario-covid-19-coronavirus-response (accessed September 16, 2021).

13. Iron K, Zagorski BM, Sykora K, Manual, Douglas G. Living and Dying in Ontario: An Opportunity for Improved Health Information ICES Investigative Report. 2008.

14. Chiu M, Lebenbaum M, Lam K, Chong N, Azimaee M, Iron K, et al. Describing the linkages of the immigration, refugees and citizenship Canada permanent resident data and vital statistics death registry to Ontario’s administrative health database. BMC Med Inform Decis Mak 2016;16. 10.1186/s12911-016-0375-3.

    https://doi.org/10.1186/s12911-016-0375-3

15. Levy AR, O’Brien BJ, Sellors C, Grootendorst P, Willison D. Coding accuracy of administrative drug claims in the Ontario Drug Benefit database. Can J Clin Pharmacol 2003;10:67–71.

16. ICES - Data Dictionary n.d. https://datadictionary.ices.on.ca/Applications/DataDictionary/Default.aspx (accessed May 31, 2021).

17. GitHub - icescentral/COVID19-Lab-Results: Parsing lab results pertaining to SARS-CoV-2 (causes COVID-19) and other respiratory viruses n.d. https://github.com/icescentral/COVID19-Lab-Results (accessed May 31, 2021).

18. Canadian Healthcare Technology - Nov. 2020 by Canadian Healthcare Technology - issuu n.d. https://issuu.com/jerryzeidenberg/docs/canadian_20healthcare_20technology_202020-08/6 (accessed May 31, 2021).

19. ICES COVID-19 Dashboard 2021. https://www.ices.on.ca/DAS/AHRQ/COVID-19-Dashboard (accessed May 3, 2021).

20. Toronto, ON; 2020.

21. Statistics Canada: Visible minority of person n.d. https://www23.statcan.gc.ca/imdb/p3Var.pl?Function=DEC&Id=45152 (accessed September 16, 2021).

22. Toronto, ON; 2020.

23. Ontario COVID-19 Science Advisory Table n.d. https://covid19-sciencetable.ca/ (accessed May 31, 2021).

24. Brown KA, Stall NM, Joh E, Allen U, Bogoch II, Buchan SA, et al. COVID-19 Vaccination Strategy for Ontario Using Age and Neighbourhood-Based Prioritization. 2021. 10.47326/ocsat.2021.02.10.1.0.

    https://doi.org/10.47326/ocsat.2021.02.10.1.0

25. Ontario Health Data Platform - OHDP n.d. https://ohdp.ca/ (accessed May 31, 2021).

26. Ontario’s COVID-19 vaccination plan |COVID-19 (coronavirus) in Ontario n.d. https://covid-19.ontario.ca/ontarios-covid-19-vaccination-plan (accessed May 31, 2021).