Impact factor (WEB OF SCIENCE - Clarivate)

2 year: 4.413 | 5 year: 5.197

Articles

External validation of the RISC, RISC-Malawi, and PERCH clinical prediction rules to identify risk of death in children hospitalized with pneumonia

Chris A Rees1, Shubhada Hooli2, Carina King3, Eric D McCollum4, Tim Colbourn5, Norman Lufesi6, Charles Mwansambo6, Marzia Lazzerini7, Shabir Ahmed Madhi8, Clare Cutland8, Marta Nunes8, Bradford D Gessner9, Sudha Basnet10, Cissy B Kartasasmita11, Joseph L Mathew12, Syed Mohammad Akram uz Zaman13, Glaucia Paranhos-Baccala14, Shinjini Bhatnagar15, Nitya Wadhwa15, Rakesh Lodha16, Satinder Aneja17, Mathuram Santosham18, Valentina S Picot19, Mariam Sylla20, Shally Awasthi21, Ashish Bavdekar22, Jean-William Pape23, Vanessa Rouzier24, Monidarin Chou25, Mala Rakoto-Andrianarivelo26, Jianwei Wang27, Pagbajabyn Nymadawa28, Philippe Vanhems29, Graciela Russomando30, Rai Asghar31, Salem Banajeh32, Imran Iqbal33, William MacLeod34, Irene Maulen- Radovan35, Greta Mino36, Samir Saha37, Sunit Singhi38, Donald M Thea39, Alexey W Clara40, Harry Campbell41, Harish Nair41, Jennifer Falconer42, Linda J Williams43, Margaret Horne43, Tor Strand44, Shamim A Qazi45, Yasir B Nisar46, Mark I Neuman47; World Health Organization PREPARE study group48

1 Division of Pediatric Emergency Medicine, Emory University School of Medicine, Children’s Healthcare of Atlanta, Atlanta, Georgia, USA
2 Section of Pediatric Emergency Medicine, Texas Children’s Hospital, Baylor College of Medicine, Houston, Texas, USA
3 Department of Global Public Health, Karolinska Institutet, Stockholm, Sweden and Institute for Global Health, University College London, London, UK
4 Global Program in Respiratory Sciences, Eudowood Division of Pediatric Respiratory Sciences, Department of Pediatrics, Johns Hopkins School of Medicine, Baltimore, USA and Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, USA
5 Institute for Global Health, University College London, London, UK
6 Ministry of Health, Lilongwe, Malawi
7 WHO Collaborating Centre for Maternal and Child Health, Institute for Maternal and Child Health IRCCS Burlo Garofolo, Trieste, Italy
8 South African Medical Research Council: Vaccines and Infectious Diseases Analytics Research Unit, School of Pathology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa; Department of Science and Technology/National Research Foundation: Vaccine Preventable Diseases Unit, University of the Witwatersrand, Johannesburg, South Africa
9 Pfizer Vaccines, Collegeville, Pennsylvania, USA
10 Center for Intervention Science in Maternal and Child Health, University of Bergen, Norway
11 Department of Child Health, Faculty of Medicine, Universitas Padjadjaran, Bandung, Indonesia
12 Pediatric Pulmonology, Postgraduate Institute of Medical Education and Research, Chandigarh, India
13 Liverpool School of Tropical Medicine, Liverpool, UK
14 Fondation Merieux, Lyon, France
15 Translational Health Science and Technology Institute, Faridabad, India
16 All India Institute of Medical Sciences, New Delhi, India
17 School of Medical Sciences & Research, Sharda University, Greater Noida, India
18 Department of International Health, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland, USA
19 Fondation Merieux, Lyon, France
20 Gabriel Touré Hospital, Department of Pediatrics, Bamako, Mali
21 King George’s Medical University, UP, Department of Pediatrics, Lucknow, India
22 KEM Hospital Pune, Department of Pediatrics, Pune, India
23 GHESKIO, Centre GHESKIO, Port au Prince, Haiti
24 GHESKIO, Department of Pediatrics, Port au Prince, Haiti
25 University of Health Sciences Faculty of Medicine, Rodolph Mérieux Laboratory, Phom Phen, Cambodia
26 Centre d’Infectiologie Charles Mérieux, Antanarivo, Madagascar
27 Chinese Academy of Medical Sciences & Peking Union, Medical College Institute of Pathogen Biology, MOH Key Laboratory of Systems Biology of Pathogens and Dr Christophe Mérieux Laboratory, Beijing, China
28 Mongolian Academy of Sciences, Academy of Medical Sciences, Ulaanbaatar, Mongolia
29 Hospices Civils de Lyon, Infection Control Unit; CIRI, Centre International de Recherche en Infectiologie, (Team PHE3ID), Université Claude Bernard Lyon, Lyon, France
30 Universidad Nacional de Asuncion, Instituto de Investigaciones en Ciencias de la Salud, San Lorenzo, Paraguay
31 Rawalpindi Medical College, Rawalpindi, Pakistan
32 Sana’a University, Sana’a, Yemen
33 Nishtar Medical College, Multan, Pakistan
34 Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, USA
35 Instituto Nactional de Pediatria Division de Investigacion Insurgentes, Mexico City, Mexico
36 Children’s Hospital Dr Francisco de Ycaza Bustamante, Head of Department, Infectious diseases, Guayaquil, Ecuador
37 Dhaka Shishu Hospital, Dhaka, Bangladesh
38 Medanta, The Medicity, Gurgaon, India
39 Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, USA
40 US Centers for Disease Control, Central American Region, Guatemala City, Guatemala
41 Centre for Global Health, Usher Institute, The University of Edinburgh, Edinburgh, Scotland
42 Institute for Global Health and Development, Queen Margaret University, Edinburgh, Scotland
43 Centre for Global Health, Usher Institute, The University of Edinburgh, Edinburgh, Scotland
44 Research Department, Innlandet Hospital Trust, Lillehammer, Norway
45 Department of Maternal, Newborn, Child and Adolescent Health (Retired), World Health Organization, Geneva, Switzerland
46 Department of Maternal, Newborn, Child and Adolescent Health and Ageing, World Health Organization, Geneva, Switzerland
47 Division of Emergency Medicine, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts, USA
48 Pneumonia Research Partnership to Assess WHO Recommendations

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Abstract

Background

Existing scores to identify children at risk of hospitalized pneumonia-related mortality lack broad external validation. Our objective was to externally validate three such risk scores.

Methods

We applied the Respiratory Index of Severity in Children (RISC) for HIV-negative children, the RISC-Malawi, and the Pneumonia Etiology Research for Child Health (PERCH) scores to hospitalized children in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set. The PREPARE data set includes pooled data from 41 studies on pediatric pneumonia from across the world. We calculated test characteristics and the area under the curve (AUC) for each of these clinical prediction rules.

Results

The RISC score for HIV-negative children was applied to 3574 children 0-24 months and demonstrated poor discriminatory ability (AUC = 0.66, 95% confidence interval (CI) = 0.58-0.73) in the identification of children at risk of hospitalized pneumonia-related mortality. The RISC-Malawi score had fair discriminatory value (AUC = 0.75, 95% CI = 0.74-0.77) among 17 864 children 2-59 months. The PERCH score was applied to 732 children 1-59 months and also demonstrated poor discriminatory value (AUC = 0.55, 95% CI = 0.37-0.73).

Conclusions

In a large external application of the RISC, RISC-Malawi, and PERCH scores, a substantial number of children were misclassified for their risk of hospitalized pneumonia-related mortality. Although pneumonia risk scores have performed well among the cohorts in which they were derived, their performance diminished when externally applied. A generalizable risk assessment tool with higher sensitivity and specificity to identify children at risk of hospitalized pneumonia-related mortality may be needed. Such a generalizable risk assessment tool would need context-specific validation prior to implementation in that setting.

Print Friendly, PDF & Email

Worldwide, pneumonia is the most common cause of mortality among children 1-59 months old [1,2]. Despite dramatic reductions in under-five mortality rates during the Millennium Development Goal era, pneumonia caused more than 800 000 pediatric deaths as recently as 2017 [2,3]. Identification of children at risk of pneumonia-related mortality is the first step in directing supportive treatments, hospitalization, and oxygen support that have the potential to reduce these deaths. Clinical prediction rules, which take into account clinical data to predict an outcome, may aid in the distribution of resources to children at greatest risk of hospitalized pneumonia-related mortality.

To date, four clinical prediction rules have been derived to identify children at risk of hospitalized pneumonia-related mortality [47]. The Respiratory Index of Severity in Children (RISC) score was developed retrospectively from >4000 hospitalized children aged 0-24 months old, with nearly 300 deaths, in South Africa in a 9-valent pneumococcal conjugate vaccine trial between 1998 and 2001 [4]. The RISC score stratified children who were human immunodeficiency virus (HIV)-positive and those who were HIV-negative. The Modified Respiratory Index of Severity in Children (mRISC) score was developed in >3500 hospitalized children aged 0-59 months with >200 deaths in Kenya from 2009 to 2012 [5]. The third score, the RISC-Malawi score, was developed retrospectively using routinely collected clinical data from a cohort of >14 000 hospitalized children 2-59 months old in Malawi, with more than 400 deaths from 2011 to 2014 [6]. Most recently, the Pneumonia Etiology Research for Child Health (PERCH) group derived a scoring system from 1800 HIV-negative children aged 1-59 months with 120 deaths from seven countries: Kenya, the Gambia, Mali, Zambia, South Africa, Thailand and Bangladesh [7]. The PERCH study aimed to determine the etiology of pneumonia among children presenting to hospitals in these regions from 2011-2014 [7].

These risk scores have not been widely validated using data from low- and middle-income settings outside sub-Saharan Africa [8,9]. The RISC score for HIV-negative children had fair discriminatory value when retrospectively applied to the PERCH and RISC-Malawi data sets, both of which were done after the pneumococcal vaccine became available and large scale prevention of mother to child transmission of HIV implementation [6,7]. The mRISC score, which used non-standard variables, to our knowledge has not been externally validated in any setting. In order for clinical prediction rules to be widely accepted and utilized, they must incorporate variables that can be feasibly collected in clinical practice.

Given the lack of published reports of their implementation, the impact of existing clinical prediction rules on reducing hospitalized pneumonia-related mortality is unclear [10]. Broad validation of existing clinical prediction rules to identify children at risk of hospitalized pneumonia-related mortality in diverse settings is the first step towards wider clinical application of useful and robust clinical prediction rules. To this end, we aimed to externally validate existing clinical prediction rules for hospitalized pneumonia-related mortality in a diverse cohort of hospitalized children from the World Health Organization’s (WHO) Pneumonia REsearch Partnership to Assess WHO REcommendations (PREPARE) study group.

METHODS

Study design

We applied the RISC, RISC-Malawi, and PERCH scores for hospitalized pneumonia-related mortality to all hospitalized children with all included parameters in the WHO PREPARE study group data set. The mRISC score includes several variables (ie, history of night sweats, dehydration, history of decreased consciousness, and history of inability to drink or breastfeed) which were not widely available in the WHO PREPARE study group data set. Thus, similar to prior work attempting to externally validate the mRISC score [6], we did not include it in our analysis. We were also unable to externally apply the RISC score for HIV-positive children as it already been applied to 152 of the 159 children who had all reported parameters in the WHO PREPARE data set [6].

Ethical considerations

This study used de-identified individual patient data from previously published studies in which ethical approval was obtained at each participating site. Additional ethical approval was obtained by the WHO ethics review committee for one study sponsored by the WHO.

Data source

The WHO PREPARE data set has been described previously [11]. Briefly, this data set includes primary, patient-level data for children 0-59 months old evaluated for pneumonia. This is from 30 diverse study groups, comprising 41 separate data sets, from over 20 low- and middle-income countries in Asia, Africa, and Latin America as well as the United States of America and Australia. Data sets were identified from a systematic review of childhood pneumonia [12]. Several additional data sets were included in PREPARE as the WHO was aware of ongoing and relevant studies. Investigators for these additional studies were invited to contribute to the PREPARE data set.

Although some data sets included older children, the cohort was restricted to children 0-59 months of age to be consistent with most other pneumonia investigations in children [1315]. We excluded community-based studies because our outcome was hospital based. We also excluded hospital-based studies that did not report survival data.

Pneumonia was defined in included studies included in the WHO PREPARE data set according to the WHO Pocket Book of Hospital Care for Children, based on the presence of age-adjusted tachypnea, lower chest indrawing, general danger signs (eg, abnormally sleepy, lethargy, central cyanosis, inability to drink, or convulsions), or signs of respiratory distress (eg, head nodding/bobbing, nasal flaring, or grunting) in children with a cough or difficulty breathing [16].

Variables

Our outcome measure was hospitalized pneumonia-related mortality during the respective study periods. Variables included in the RISC, RISC-Malawi, and PERCH scores are found in Table 1. The RISC score for HIV-negative children had a range of -2 to 6 points [4]. All variables included in the original RISC score for HIV-negative children were included in this external application. The RISC score assigned 3 points to HIV-negative children with SpO2≤90%. However, if a child’s SpO2 was >90%, chest indrawing became a part of the scoring system. If the SpO2 was ≤90%, chest indrawing was not included in the score assignment. Though originally derived among children 0-24 months old, we also applied the RISC score for HIV-negative children to children 0-59 months old who had all necessary parameters available as a supplement.

Table 1.  Summary of variables included in the Respiratory Index of Severity in Children (RISC) [4], RISC-Malawi [6], and Pneumonia Etiology Research for Child Health (PERCH) [7] hospitalized pneumonia-related mortality prediction scores

*Deep breathing not available in WHO PREPARE data set.

†Weight for age z-score used in the external application.

The RISC-Malawi score had a range of -1 to 17 using weight-for-age z-score (WAZ) and a range of -2 to 23 points using mid-upper arm circumference (MUAC) [6]. As MUAC was only reported in two studies that met all inclusion criteria within the WHO PREPARE data set [17,18], we used the RISC-Malawi score with WAZ, applied to children aged 2-59 months, for our external application. The RISC-Malawi score using MUAC analysis is included as a supplement. The WHO PREPARE data set contains the RISC-Malawi development data set, and therefore these data were excluded from the external validation of the RISC-Malawi score [6].

The PERCH score had a range of -1 to 17 [7]. Its primary outcome was hospitalized pneumonia-related mortality or death within 7 days of hospital discharge. The PREPARE data set does not include the variable “deep breathing”, which was included in the PERCH study, so this was not included in our analysis. Therefore, the maximum PERCH score in our external validation was 12 instead of 17. Furthermore, the PERCH score included the parameter of observed cough. For our external application, we included history of cough in place of observed cough. We used the variable unconsciousness in place of unresponsiveness as the definitions of both of these included descriptions such as unconsciousness, unresponsiveness, lethargy, and abnormally sleepy. In lieu of weight-for-height, we used WAZ to measure children’s nutritional status as follows: weight-for-height z-score<-3 corresponded to WAZ<-3 for severe acute malnutrition, weight-for-height z-score≥–3 to<–2 corresponded to WAZ of -3 to -2 for moderate acute malnutrition, and weight-for-height z-score≥-2 corresponded to WAZ>-2 for normal weight.

Data analysis

We determined the accuracy of the RISC, RISC-Malawi, and PERCH scores on hospitalized children 0-59 months old in the WHO PREPARE data set. We calculated the case fatality ratio (CFR), as the number of deaths/number of patients, and the percentage of patients who were correctly classified as true positive and true negative cases among all children at each cut point for each score. We calculated the sensitivity, specificity, and positive and negative likelihood ratios (LR) of each score at ≥ each specified cut point as defined in the scores. We created receiver operating characteristic (ROC) curves for each risk score. Based on published standards, we used the following scale to qualify the discriminatory ability of each score: area under the curve (AUC)≥0.90 for “excellent discrimination”, AUC 0.80 to 0.89 for “good discrimination”, AUC = 0.70-0.79 for “fair discrimination”, and “poor discrimination” for AUC<0.70 [19,20].

We conducted a complete case analysis and excluded cases with missing data for any parameter. In order to accurately identify children at low-risk of hospitalized pneumonia-related mortality, we aimed to report maximum sensitivity without sacrificing specificity. We, therefore, report risk score cut points with the sensitivity of 60%-80% with concurrent specificity of at least 40%. We present risk predictiveness curves, to show the cumulative percentage of children at risk of hospitalized pneumonia-related mortality by their predicted risk. We report test characteristics and 95% confidence intervals (CI) for the +LRs and the -LRs. All analyses were conducted using Stata version 14.2 (Stata-Corp, College Station, TX, USA).RESULTS

Study population

Of the 41 separate data sets in the PREPARE study, there were 26 hospital-based studies and 15 community-based studies (Figure 1). Of the 26 hospital-based studies in the WHO PREPARE data set, 24 studies included data on patients’ survival status through hospitalization, with 228 460 patients aged 0-59 months old who were evaluated for pneumonia. Among these, there were 8820 (3.9%) hospitalized pneumonia-related deaths. We were able to apply the RISC score for HIV-negative children 0-24 months old to 3574 children from five studies [6,17,18,21,22], the RISC-Malawi score to 17 864 children 2-59 months old from 10 studies [17,18,2128], and the PERCH score to 732 children 1-59 months old from two studies [17,21] (Figure 1). A total of 732 children had data available for all three scores [17,21]. Characteristics of individual included studies are found in Table 2. The CFR in the included studies ranged from 0.9%-10.9%.

Figure 1.  Selection of children aged 0-59 months from hospital-based studies included in the external application of the Respiratory Index of Severity in Children (RISC), RISC-Malawi, and Pneumonia Etiology Research for Child Health (PERCH) scores. aThe sum of patients included in the 4 individual validation cohorts does not total 41 365 as the validation each of the 4 scores required that every component of the score be available within the data set. bExcluding the parent data set from which the RISC-Malawi score was derived. cExcluding 610 children less than 2 months of age. dExcluding 105 children less than 2 months of age.

Table 2.  Characteristics of studies included in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set

*WHO-defined pneumonia: presence of age-specific fast breathing, lower chest indrawing, or general danger signs in children with a cough or difficulty

†Cough and/or difficulty breathing, and central cyanosis or inability to drink.

RISC score for HIV-negative children

The numbers of patients who had each parameter of the RISC score for HIV-negative children with corresponding CFRs are found in Table 3. Of the parameters in the RISC score for HIV-negative children 0-24 months old, children who refused to feed at presentation had the highest CFR and children who had presented with wheeze had the lowest CFR. The RISC score for HIV-negative children 0-24 months old had poor discriminatory ability in identifying children at risk of hospitalized pneumonia-related mortality (Figure 2). The CFR at a RISC score of 2 out of 6 for HIV-negative children 0-24 months old was 0.8%. Using a score of ≥2, the RISC score for HIV-negative children 0-24 months old had 74.1% sensitivity, 41.4% specificity, +LR of 1.26 (95% CI 1.05-1.44), and -LR of 0.64 (95% CI 0.45-1.01) (Table S1 in the Online Supplementary Document). The RISC score for HIV-negative children performed similarly when applied to 4061 children aged 0-59 months with a score of 2 out of 6 (Table S2 and Table S3 in the Online Supplementary Document).

Table 3.  Number of patients, deaths, and case fatality ratio of parameters used in the Respiratory Index of Severity in Children (RISC) score for HIV-negative children 0-24 mo old in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set (n = 3574)

Figure 2. Receiver Operating Characteristic (ROC) curve for the Respiratory Index of Severity in Children (RISC) score for HIV-negative children 0-24 months old, RISC-Malawi weight-for-age (WAZ) score for children 2-59 months old, and Pneumonia Etiology Research for Child Health (PERCH) score for HIV-negative children 1-59 months old for hospitalized pneumonia-related mortality.

RISC-Malawi Score

We applied the RISC-Malawi score using WAZ to 17 864 children 2-59 months old in the WHO PREPARE data set. Of the parameters in the RISC-Malawi score, patients who had documented unconsciousness at presentation had the highest CFR, and children with SpO2≥93% had the lowest CFR (Table 4). The RISC-Malawi score had fair discriminatory value in the identification of children at risk of hospitalized pneumonia-related mortality (Figure 2). At a score of ≥5 out of 17, the RISC-Malawi score had 75.1% sensitivity, 67.9% specificity, +LR of 2.27 (95% CI = 2.17-2.37), and -LR of 0.37 (95% CI = 0.33-0.42) in identifying children at risk of hospitalized pneumonia-related mortality (Table S4 in the Online Supplementary Document). Using a cut-point score of ≥6 out of 17, the RISC-Malawi score using WAZ had 61.6% sensitivity, 75.5% specificity, +LR of 2.51 (95% CI = 2.37-2.66), and -LR of 0.51 (95% CI = 0.47-0.55) (Table S4 in the Online Supplementary Document).

Table 4.  Number of patients, deaths, and case fatality ratio for parameters in the Respiratory Index of Severity in Children (RISC)-Malawi score using weight-for-age z-score (WAZ) for children 2-59 mo old in the PREPARE data set (n = 17 864)

When applying the RISC-Malawi MUAC score to 2251 patients, the parameter SpO2<90% demonstrated the highest CFR (Table S5 in the Online Supplementary Document). Though the +LR and -LR were slightly lower with MUAC included instead of WAZ, overall the RISC-Malawi score performed similarly with MUAC included as the parameter for malnutrition (Table S6 in the Online Supplementary Document).

PERCH score for HIV-negative children

We applied the PERCH score to 732 children 1-59 months old in the WHO PREPARE data set. Of the parameters included in the PERCH score, patients who had WAZ<-3 had the highest CFR. Children with unresponsiveness had the lowest CFR (Table 5). The PERCH score had poor discriminatory value in identifying children at risk of hospitalized pneumonia-related mortality (Figure 2). The CFR at a PERCH score of 5 was 0.6%. A PERCH score of ≥5 had 62.5% sensitivity, 29.6% specificity, +LR of 0.89 (95% CI = 0.61-1.30), and -LR of 1.27 (95% CI = 0.67-2.41) (Table S7 in the Online Supplementary Document).

Table 5.  Number of patients, deaths, and case fatality ratio by parameter used in the Pneumonia Etiology Research for Child Health (PERCH) study score for HIV-negative children 1-59 mo old in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set (n = 732)

Risk predictiveness curves

Risk predictiveness curves demonstrated that the majority of children were low risk for mortality (Figure 3).

Figure 3.  Risk predictiveness curves of the Respiratory Index of Severity in Children (RISC) score for HIV-negative children 0-24 months old (A), RISC-Malawi weight-for-age (WAZ) score for children 2-59 months old (B), and Pneumonia Etiology Research for Child Health (PERCH) score for HIV-negative children 1-59 months old (C) for hospitalized pneumonia-related mortality.

DISCUSSION

Clinical prediction rules previously derived among children in sub-Saharan Africa, Thailand, and Bangladesh performed sub-optimally in identifying hospitalized pneumonia-related mortality when externally applied to a diverse population of children from 19 countries. Both the RISC score for HIV-negative children and the PERCH score had poor discriminatory value when externally applied to the PREPARE data set, and the RISC-Malawi score had fair discriminatory value in identifying children at risk of hospitalized pneumonia-related mortality.

Among HIV-negative children, we observed that the RISC score had poor discriminatory value in identifying children at risk of hospitalized pneumonia-related mortality in both children 0-24 and 0-59 months old. This is in contrast to the external application of the RISC score for HIV-negative among >14 000 children in Malawi and the external application in the PERCH data set including >1800 children from seven countries in which RISC score had fair discriminatory value among all children and among HIV-exposed children [6,7]. The poor performance of the RISC score in our external application may be due to having more diverse patient populations, including differing rates and types of comorbidities among children outside sub-Saharan Africa. Although 68% of patients included in our external application were located in sub-Saharan Africa, these studies generally have lower childhood mortality rates than the original patient group. Given the challenges we found in accessing reliable routine data on HIV-status in children, changes in HIV prevention and management in sub-Saharan Africa [29], and assumed changes in underlying pneumonia etiology since widespread pneumococcal vaccine introduction [30], poor to fair external performance was not necessarily surprising.

Though the PERCH score derivation included multiple countries, unlike the other scores evaluated, it had the poorest discriminatory value in identifying children at risk of hospitalized pneumonia-related mortality. This may be partly explained by our inability to include the variable “deep breathing” or the substitution of WAZ in place of weight-for-height z-score. Deep breathing is not a standard variable in the WHO screening for pneumonia. Similar to the mRISC score, which has not been externally validated, this highlights the challenge of including variables that are not included in routine guidelines in practical clinical prediction rules. While such inclusion may improve local performance, their ability to be pragmatically scaled or validated may be limited. Furthermore, our substitution of WAZ for weight-for-height z-score may also limit the external validation of the PERCH score as weight-for-height z-score may more accurately account for both present and past nutritional status [31]. Further validation studies using the precise variables included in the PERCH score across a variety of settings are warranted.

Among the clinical prediction rules externally applied in our study, the RISC-Malawi score had the most discriminatory value. However, the AUC for the RISC-Malawi score demonstrated only fair discriminatory value. The improved performance of the RISC-Malawi score may be due to fewer physical examination findings included as parameters in this score. Variation in inter-observer agreement of physical examination findings could lead to classification errors that would ultimately under-estimate the effect of a clinical prediction tool [32,33]. Moreover, malaria and HIV, though endemic in Malawi, were not incorporated into the score due to underreporting of these variables [6]. As HIV and malaria are common in Malawi among children [34,35], it is possible that the exclusion of these variables may have driven mortality risk as an unmeasured confounding parameter. Lastly, we were able to apply the RISC-Malawi score to over 17 000 children, which suggests it may be the clinical prediction rule that allows for widest dissemination based on the use of data routinely collected in clinical practice across a variety of settings.

Although clinical prediction rules tend to underperform when applied externally to populations in which they were derived, none of the clinical prediction rules externally applied here demonstrated excellent discriminatory value. Thus, clinical prediction rules for hospitalized pneumonia-related mortality may not be confidently applied in settings beyond where they were derived. Some of the underperformance of these clinical prediction rules in our external validation may be due to varying resources such as high-flow nasal cannula, non-invasive ventilation, or mechanical ventilation and perhaps limited oxygen and pulse oximetry in some settings included in the WHO PREPARE data set. Furthermore, variations in quality of care, diagnostic capabilities, region-specific pathogens implicated in childhood pneumonia, and antibiotic use across the 19 countries included in our analysis may have contributed to the underperformance of these clinical prediction rules.

Age <12 months, chest indrawing, grunting, respiratory rate >70 breaths/min, hepatomegaly, acute moderate and severe malnutrition, and the presence of moderate or severe pallor have been shown to be independently associated with pneumonia-related mortality among children and may be candidate predictors in future risk assessment tools [36,37]. Both the RISC and RISC-Malawi scores demonstrated that the presence of wheezing was associated with lower mortality rates among children, likely suggestive of the presence of a viral etiology, such as bronchiolitis [38]. Moreover, serum lactate and biomarkers such as C-reactive protein and procalcitonin may be associated with severe outcomes in childhood pneumonia [3942] and may be considered for implementation in future mortality risk assessment tools in settings where these tests are available. However, such biomarkers may not be widely available, could prove cost prohibitive, and involve mildly invasive blood draws.

Limitations

Though this is the first broad external validation outside of sub-Saharan Africa of the RISC, RISC-Malawi, and PERCH scores, our analysis is subject to limitations. First, the PREPARE data set includes data from studies around the world that were conducted for varying reasons and not necessarily for the external application of these hospitalized pneumonia-related mortality clinical prediction rules. This led to missing variables for many children which precluded our ability to include children with missing data in our external application of these risk scores. As a result, the ROC curves for each score were applied to different patient populations with varying sample sizes. We were not able to apply the RISC score for HIV-positive children to this diverse data set due to lack of reporting of HIV status. This may be an artifact of the enrollment protocols of the studies included in our data set. A large proportion of patients included in our external application of these scores were from Malawi and South Africa, which may have over- or under-estimated the performance of these scores in settings outside of sub-Saharan Africa. Furthermore, we did not control for patient skin color or altitude, which may affect the accuracy of pulse oximetry [4345].

In terms of the PERCH score, we were unable to externally validate this tool as its developers intended because the WHO PREPARE data set did not include a variable for deep breathing, which may have led to some mis-calibration of our external application of the PERCH risk score. Furthermore, we were only able to apply the PERCH score to 732 children who had complete data for the variables included. Additionally, we only analyzed hospitalized pneumonia-related mortality as post-discharge mortality was not available in the PREPARE data set and the PERCH score predicts hospital and up to 7-day post-discharge mortality. However, post-discharge mortality accounted for only 7/120 (5.8%) of the deaths in the PERCH study. Lastly, we were unable to assess for collinearity of other variables that may have contributed to hospitalized pneumonia-related mortality, as children admitted to hospitals in resource-limited settings often carry more than one diagnosis concomitantly, which could have driven mortality.

CONCLUSIONS

In a large external application of the RISC, RISC-Malawi, and PERCH scores, a substantial number of children were misclassified for their risk of hospitalized pneumonia-related mortality. The RISC score for HIV-negative children and the PERCH score had poor discriminatory value. The RISC-Malawi score had fair discriminatory value in identifying children at risk of hospitalized pneumonia-related mortality. Further development of a risk assessment tool with greater sensitivity and specificity to identify children at risk of hospitalized pneumonia-related mortality may be warranted. However, careful consideration is needed on whether clinical prediction rules for hospitalized pneumonia-related mortality should aim to be region-specific, considering local epidemiology and resources, or universal. Finally, risk assessment tools must be reassessed over time particularly with the advent of new interventions such as vaccines and medications, and as health systems strengthen and are more able to provide pulse oximetry measurement, supplemental oxygen, and antibiotics.

Additional material

Online Supplementary Document

Acknowledgements

Disclaimer: The expressed views and opinions do not necessarily represent the policies of the World Health Organization.

[1] Funding: The study was funded by the Bill & Melinda Gates Foundation (#INV-007927) through a grant to the World Health Organization. The funders had no role in the study design or in the collection, analysis, or interpretation of the data. The funders did not write the report and had no role in the decision to submit the paper for publication.

[2] Authorship contributions: SAQ secured the funding. CAR, MIN, YBN, and SAQ conceptualized and designed the study, interpreted the data, and wrote the first draft of the manuscript. YBN verified the underlying data and conducted the statistical analyses. TC, EDM, CK, NL, CM, SH, ML, SAM, CC, MN, BDG, SBasnet, CBK, JM, SYMAZ, GP-B, SBhatnagar, NW, RL, SAneja, MSantosham, VSP, MSylla, SAwasthi, AB, J-WP, VP, MC, MR-A, JW, PN, PV, GR, RA, SBanajeh, II, WM, IM-R, GM, SSaha, SSinghi, DMT, AWC, HC, HN, JF, LJW, MH, and TS oversaw data collection, verified the underlying data, assisted with the interpretation of the data, and reviewed and provided input to the final draft. CAR had final responsibility for the decision to submit for publication.

[3] Competing interests: The authors have completed the ICMJE Declaration of Interest Form (available upon request from the corresponding author), and declare no conflicts of interest. YBN is staff member of the World Health Organization.

references

[1] L Liu, S Oza, D Hogan, J Perin, I Rudan, and JE Lawn. Global, regional, and national causes of child mortality in 2000-13, with projections to inform post-2015 priorities: An updated systematic analysis. Lancet. 2015;385:430-40. DOI: 10.1016/S0140-6736(14)61698-6. [PMID:25280870]

[2] . Quantifying risks and interventions that have affected the burden of lower respiratory infections among children younger than 5 years: an analysis for the Global Burden of Disease Study 2017. Lancet Infect Dis. 2020;20:60-79. DOI: 10.1016/S1473-3099(19)30410-4. [PMID:31678026]

[3] D You, L Hug, S Ejdemyr, P Idele, D Hogan, and C Mathers. Global, regional, and national levels and trends in under-5 mortality between 1990 and 2015, with scenario-based projections to 2030: A systematic analysis by the un Inter-Agency Group for Child Mortality Estimation. Lancet. 2015;386:2275-86. DOI: 10.1016/S0140-6736(15)00120-8. [PMID:26361942]

[4] C Reed, SA Madhi, KP Klugman, L Kuwanda, JF Ortiz, and L Finelli. Development of the respiratory index of severity in children (risc) score among young children with respiratory infections in South Africa. PLoS One. 2012;7:e27793. DOI: 10.1371/journal.pone.0027793. [PMID:22238570]

[5] GO Emukule, M McMorrow, C Ulloa, S Khagayi, HN Njuguna, and D Burton. Predicting mortality among hospitalized children with respiratory illness in Western Kenya, 2009-2012. PLoS One. 2014;9:e92968. DOI: 10.1371/journal.pone.0092968. [PMID:24667695]

[6] S Hooli, T Colbourn, N Lufesi, A Costello, B Nambiar, and S Thammasitboon. Predicting hospitalised paediatric pneumonia mortality risk: An external validation of RISC and mRISC, and local tool development (RISC-Malawi) from Malawi. PLoS One. 2016;11:e0168126. DOI: 10.1371/journal.pone.0168126. [PMID:28030608]

[7] KE Gallagher, MD Knoll, C Prosperi, HC Baggett, WA Brooks, and DR Feikin. The Predictive Performance of a Pneumonia Severity Score in Human Immunodeficiency Virus-negative Children Presenting to Hospital in 7 Low- and Middle-income Countries. Clin Infect Dis. 2020;70:1050-7. [PMID:31111870]

[8] A Arbo, D Lovera, and C Martínez-Cuellar. Mortality Predictive Scores for Community-Acquired Pneumonia in Children. Curr Infect Dis Rep. 2019;21:10 DOI: 10.1007/s11908-019-0666-9. [PMID:30834468]

[9] M Ogero, RJ Sarguta, L Malla, J Aluvaala, A Agweyu, and M English. Prognostic models for predicting in-hospital paediatric mortality in resource-limited countries: a systematic review. BMJ Open. 2020;10:e035045. DOI: 10.1136/bmjopen-2019-035045. [PMID:33077558]

[10] KV Deardorff, ED McCollum, and AS Ginsburg. Pneumonia risk stratification scores for children in low-resource settings: A systematic literature review. Pediatr Infect Dis J. 2018;37:743-8. DOI: 10.1097/INF.0000000000001883. [PMID:29278608]

[11] CA Rees, S Basnet, A Gentile, BD Gessner, CB Kartasasmita, and M Lucero. An analysis of clinical predictive values for radiographic pneumonia in children. BMJ Glob Health. 2020;5:e002708. DOI: 10.1136/bmjgh-2020-002708. [PMID:32792409]

[12] H Nair, EAF Simões, I Rudan, BD Gessner, E Azziz-Baumgartner, and JSF Zhang. Global and regional burden of hospital admissions for severe acute lower respiratory infections in young children in 2010: A systematic analysis. Lancet. 2013;381:1380-90. DOI: 10.1016/S0140-6736(12)61901-1. [PMID:23369797]

[13] ED McCollum, SP Brown, E Nkwopara, T Mvalo, S May, and AS Ginsburg. Development of a prognostic risk score to aid antibiotic decision-making for children aged 2-59 months with World Health Organization fast breathing pneumonia in Malawi: An Innovative Treatments in Pneumonia (ITIP) secondary analysis. PLoS One. 2019;14:e0214583. DOI: 10.1371/journal.pone.0214583. [PMID:31220085]

[14] L Liu, Y Chu, S Oza, D Hogan, J Perin, and DG Bassani. National, regional, and state-level all-cause and cause-specific under-5 mortality in India in 2000-15: a systematic analysis with implications for the Sustainable Development Goals. Lancet Glob Health. 2019;7:e721-34. DOI: 10.1016/S2214-109X(19)30080-4. [PMID:31097276]

[15] AS Ginsburg, T Mvalo, E Nkwopara, ED McCollum, M Phiri, and R Schmicker. Amoxicillin for 3 or 5 Days for Chest-Indrawing Pneumonia in Malawian Children. N Engl J Med. 2020;383:13-23. DOI: 10.1056/NEJMoa1912400. [PMID:32609979]

[16] World Health Organization. Pocket book of hospital care for children: guidelines for the management of common childhood illnesses 2nd Edition. Available: https://apps.who.int/iris/bitstream/handle/10665/81170/9789241548373_eng.pdf?sequence=1. Published 2013. Accessed: 1 December 2020.

[17] JL Mathew, S Singhi, P Ray, E Hagel, S Saghafian-Hedengren, and A Bansal. Etiology of community acquired pneumonia among children in India: Prospective, cohort study. J Glob Health. 2015;5:050418. DOI: 10.7189/jogh.05.020418. [PMID:26528392]

[18] T Bénet, V Sánchez Picot, M Messaoudi, M Chou, T Eap, and J Wang. Microorganisms Associated with Pneumonia in Children <5 Years of Age in Developing and Emerging Countries: The GABRIEL Pneumonia Multicenter, Prospective, Case-Control Study. Clin Infect Dis. 2017;65:604-12. DOI: 10.1093/cid/cix378. [PMID:28605562]

[19] MW Bijlsma, MC Brouwer, PM Bossuyt, MW Heymans, AVD Ende, and MWT Tanck. Risk scores for outcome in bacterial meningitis: Systematic review and external validation study. J Infect. 2016;73:393-401. DOI: 10.1016/j.jinf.2016.08.003. [PMID:27519619]

[20] MP Muller, G Tomlinson, TJ Marrie, P Tang, A McGeer, and DE Low. Can routine laboratory tests discriminate between severe acute respiratory syndrome and other causes of community-acquired pneumonia? Clin Infect Dis. 2005;40:1079-86. DOI: 10.1086/428577. [PMID:15791504]

[21] S Basnet, PS Shrestha, A Sharma, M Mathisen, R Prasai, and N Bhandari. A randomized controlled trial of zinc as adjuvant therapy for severe pneumonia in young children. Pediatrics. 2012;129:701-8. DOI: 10.1542/peds.2010-3091. [PMID:22392179]

[22] Clara WC. Unpublished Data from Chiriqui Province, Panama.

[23] BD Gessner, A Sutanto, M Linehan, IGG Djelantik, T Fletcher, and IK Gerudug. Incidences of vaccine-preventable Haemophilus influenzae type b pneumonia and meningitis in Indonesian children: hamlet-randomised vaccine-probe trial. Lancet. 2005;365:43-52. DOI: 10.1016/S0140-6736(04)17664-2. [PMID:15643700]

[24] KP Klugman, SA Madhi, RE Huebner, R Kohberger, N Mbelle, and N Pierce. A Trial of a 9-Valent Pneumococcal Conjugate Vaccine in Children with and Those without HIV Infection. N Engl J Med. 2003;349:1341-8. DOI: 10.1056/NEJMoa035060. [PMID:14523142]

[25] R Asghar, S Banajeh, J Egas, P Hibberd, I Iqbal, and M Katep-Bwalya. Chloramphenicol versus ampicillin plus gentamicin for community acquired very severe pneumonia among children aged 2-59 months in low resource settings: Multicentre randomised controlled trial (SPEAR study). BMJ. 2008;336:80-4. DOI: 10.1136/bmj.39421.435949.BE. [PMID:18182412]

[26] N Wadhwa, A Chandran, S Aneja, R Lodha, SK Kabra, and MK Chaturvedi. Efficacy of zinc given as an adjunct in the treatment of severe and very severe pneumonia in hospitalized children 2-24 mo of age: A randomized, double-blind, placebo-controlled trial. Am J Clin Nutr. 2013;97:1387-94. DOI: 10.3945/ajcn.112.052951. [PMID:23636236]

[27] FT Cutts, SM Zaman, G Enwere, S Jaffar, OS Levine, and JB Okoko. Efficacy of nine-valent pneumococcal conjugate vaccine against pneumonia and invasive pneumococcal disease in The Gambia: randomised, double-blind, placebo-controlled trial. Lancet. 2005;365:1139-46. DOI: 10.1016/S0140-6736(05)71876-6. [PMID:15794968]

[28] Wulandari DA, Sudarwati S, Tirtosudiro MA, Nataprawira HM, Kartasasmita CB. Risk Factors for Mortality in Children Hospitalized with Pneumonia in Hasan Sadikin Hospital. In: Asian Paediatric Pulmonology Society. Beijing, China; 2018.

[29] . Global; regional; and national incidence; prevalence; and mortality of HIV; 1980-2017; and forecasts to 2030; for 195 countries and territories: a systematic analysis for the Global Burden of Diseases; Injuries; and Risk Factors Study 2017. Lancet HIV. 2019;6:e831-59. DOI: 10.1016/S2352-3018(19)30196-1. [PMID:31439534]

[30] . Causes of severe pneumonia requiring hospital admission in children without HIV infection from Africa and Asia: the PERCH multi-country case-control study. Lancet. 2019;394:757-79. DOI: 10.1016/S0140-6736(19)30721-4. [PMID:31257127]

[31] S Sachdeva, P Dewan, D Shah, RK Malhotra, and P Gupta. Mid-upper arm circumference v. weight-for-height Z-score for predicting mortality in hospitalized children under 5 years of age. Public Health Nutr. 2016;19:2513-20. DOI: 10.1017/S1368980016000719. [PMID:27049813]

[32] TA Florin, L Ambroggio, C Brokamp, MS Rattan, EJ Crotty, and A Kachelmeyer. Reliability of Examination Findings in Suspected Community-Acquired Pneumonia. Pediatrics. 2017;140:e20170310. DOI: 10.1542/peds.2017-0310. [PMID:28835381]

[33] MW Sjoding, CR Cooke, TJ Iwashyna, and TP Hofer. Acute Respiratory Distress Syndrome Measurement Error. Potential Effect on Clinical Study Results. Ann Am Thorac Soc. 2016;13:1123-8. DOI: 10.1513/AnnalsATS.201601-072OC. [PMID:27159648]

[34] UNAIDS. UNAIDS: Country Profile Malawi. Available: https://www.unaids.org/en/regionscountries/countries/malawi. Published 2019. [Accessed September 27, 2020].

[35] AS Muula, E Rudatsikira, S Siziya, and RH Mataya. Estimated financial and human resources requirements for the treatment of malaria in Malawi. Malar J. 2007;6:168 DOI: 10.1186/1475-2875-6-168. [PMID:18093319]

[36] AM Demers, P Morency, F Mberyo-Yaah, S Jaffar, C Blais, and P Somse. Risk factors for mortality among children hospitalized because of acute respiratory infections in Bangui, Central African Republic. Pediatr Infect Dis J. 2000;19:424-32. DOI: 10.1097/00006454-200005000-00007. [PMID:10819338]

[37] A Agweyu and RJ Lilford. English, Clinical Information Network Author Group. Appropriateness of clinical severity classification of new WHO childhood pneumonia guidance: a multi-hospital, retrospective, cohort study. Lancet Glob Health. 2018;6:e74-83. DOI: 10.1016/S2214-109X(17)30448-5. [PMID:29241618]

[38] FS Dawood, AM Fry, D Goswami, A Sharmeen, K Nahar, and BA Anjali. Incidence and characteristics of early childhood wheezing, Dhaka, Bangladesh, 2004-2010. Pediatr Pulmonol. 2016;51:588-95. DOI: 10.1002/ppul.23343. [PMID:26613245]

[39] B Ramakrishna, SM Graham, A Phiri, L Mankhambo, and T Duke. Lactate as a predictor of mortality in Malawian children with WHO-defined pneumonia. Arch Dis Child. 2012;97:336-42. DOI: 10.1136/archdischild-2011-300920. [PMID:22267369]

[40] MM Higdon, T Le, KL O’Brien, DR Murdoch, C Prosperi, and HC Baggett. Association of C-Reactive Protein With Bacterial and Respiratory Syncytial Virus-Associated Pneumonia Among Children Aged <5 Years in the PERCH Study. Clin Infect Dis. 2017;64:Suppl_3S378-86. DOI: 10.1093/cid/cix150. [PMID:28575375]

[41] A Galetto-Lacour, G Alcoba, KM Posfay-Barbe, M Cevey-Macherel, M Gehri, and MM Ochs. Elevated inflammatory markers combined with positive pneumococcal urinary antigen are a good predictor of pneumococcal community-acquired pneumonia in children. Pediatr Infect Dis J. 2013;32:1175-9. DOI: 10.1097/INF.0b013e31829ba62a. [PMID:23694836]

[42] J van De Maat, D Nieboer, M Thompson, M Lakhanpaul, H Moll, and R Oostenbrink. Can clinical prediction models assess antibiotic need in childhood pneumonia? A validation study in paediatric emergency care. PLoS One. 2019;14:e0217570. DOI: 10.1371/journal.pone.0217570. [PMID:31194750]

[43] MW Sjoding, RP Dickson, TJ Iwashyna, SE Gay, and TS Valley. Racial Bias in Pulse Oximetry Measurement. N Engl J Med. 2020;383:2477-8. DOI: 10.1056/NEJMc2029240. [PMID:33326721]

[44] EE Foglia, RK Whyte, A Chaudhary, A Mott, J Chen, and KJ Propert. The Effect of Skin Pigmentation on the Accuracy of Pulse Oximetry in Infants with Hypoxemia. J Pediatr. 2017;182:375-377.e2. DOI: 10.1016/j.jpeds.2016.11.043. [PMID:27939107]

[45] S Ucrós, CM Granados, JA Castro-Rodríguez, and CM Hill. Oxygen Saturation in Childhood at High Altitude: A Systematic Review. High Alt Med Biol. 2020;21:114-25. DOI: 10.1089/ham.2019.0077. [PMID:32239983]

Correspondence to:
Yasir B. Nisar
Department of Maternal, Newborn, Child and
Adolescent Health
World Health Organization
20 Avenue Appia 1211, 27
Geneva
Switzerland
[email protected]