Recommendations for refining key maternal health policy and finance indicators to strengthen a framework for monitoring the Strategies toward Ending Preventable Maternal Mortality (EPMM)

The Millennium Development Goals (MDGs) comprised 8 goals, 18 targets, and 48 indicators; the Sustainable Development Goal (SDG) framework includes 17 goals, 169 targets, and 232 indicators. Similar expansion is seen in maternal newborn health (MNH) measurement [1]. In a 2019 systematic review, 1445 unique MNH indicators were identified [2]. The burden associated with proliferating indicators necessitates greater focus on quality for those chosen for monitoring.

Various sources propose criteria for evaluating measure quality, which generally include importance, relevance, utility, validity, feasibility, and distinctiveness [3–5]. How clearly the indicator definition captures the underlying construct for measurement, and whether its metadata are constructed so it is reliable, valid, and interpretable [2,6], further influence indicator quality. Effectiveness, the degree to which any objective is achieved [7], can only be evaluated when objectives are: a) clearly defined a priori and b) quantifiable [8]. Poorly defined measures are thus unlikely to be valid, or effective in tracking changes, with the ultimate goal of driving improvement. Therefore, it is concerning that many MNH measures in use lack scientific soundness as evidenced by a complete definition; demonstrated validity, reliability, and feasibility; and plausible application [2].

Health policy indicators are systematically evaluated less frequently than clinical indicators, despite growing awareness of the need for robust data to drive health policy decision-making, system improvement, and accountability to address upstream determinants of maternal health and survival. Efforts to assess validity of indicators designed for monitoring health system and policy factors are challenged by a lack of systematic approaches [9].

Indicators to monitor maternal health financing present particular measurement challenges. A major focus of the SDGs is closing financing gaps to achieve a “grand convergence” [10] in MNH outcomes between countries across income levels, through adequate domestic resource allocation and harmonization in Official Development Assistance (ODA). The Global Financing Facility (GFF) was established in 2015 to reform RMNCH financing to meet the SDGs, and now includes 36 countries [11–13]. Universal Health Coverage (UHC), with prioritization of MCH, is considered key to achieving SDG 3.1 [14]. The former UN Secretary General noted achieving UHC depends upon effective measures for tracking health financing and spending, both governmental and out-of-pocket [15]. However, most health financing measures are not optimized for MNH monitoring. Measurement experts have reported interest in MNH health finance indicators [9]; however, many common economic indicators are not disaggregated to allow tracking of maternal health financing, not routinely collected or reported, or not available in the public domain, among other issues [16–19].

In 2015, WHO released the Strategies toward Ending Preventable Maternal Mortality (EPMM) (EPMM Strategies) [20] to serve as a global framework for maternal health during the SDGs. Subsequently, indicators tailored to the report’s 11 Key Themes [20] were identified for a monitoring framework comprising: a core set maternal health indicators for global reporting [21], and a menu of indicators for national monitoring to track a broad range of social, political, economic and health system determinants of maternal health and survival [22]. The latter were selected through a five-round modified Delphi process to identify the 1-3 strongest available measures to monitor progress toward each EPMM Key Theme. The selection criteria utilized in this process appear in Table 1.

Table 1.  EPMM indicator selection criteria

In 2017, the Women and Health Initiative (W&HI) at the Harvard T.H. Chan School of Public Health initiated the Improving Maternal Health Measurement (IMHM) Project, whose primary aim is to strengthen indicators for monitoring the EPMM Strategies. In December 2018, the W&HI convened technical experts in maternal health policy (Consultation 1) and maternal health financing (Consultation 2) to address problems in a selection of these measures. The specific aim was a set of recommendations to improve the validity and utility of selected measures for monitoring key themes of the EPMM Strategies. As many EPMM indicators bridge domains, a secondary aim was intersectoral coordination to improve measurement capacity overall. This paper summarizes the recommendations that emanated from these deliberations.Participants

We purposively invited experts in MNH measure development and the topical areas covered by the selected indicators. We specifically included an expert affiliated with the data custodian agency whenever possible, and measurement experts with experience implementing each indicator. In total, forty participants from thirteen countries (Ghana, USA, Brazil, UK, Argentina, Switzerland, Kenya, Bangladesh, Nigeria, Germany, Belgium, Congo-Brazzaville, and India) attended two consecutive domain-specific technical consultations; some participants attended both. There were twenty-six participants in Consultation 1 and twenty-seven in Consultation 2 (Appendix S1 of the Online Supplementary Document).Conference structure

Selection of indicators for strengthening

Five maternal health policy indicators (Consultation 1) and five maternal health financing indicators (Consultation 2) were included, presented with full metadata in Table 2. From 2017-2018, stakeholders were queried regarding EPMM indicators in need of strengthening through the IMHM Project, during a review of EPMM indicator use in 20 countries, a global stakeholder meeting to prioritize EPMM indicators for validation research, and a poll of IMHM Project advisors. Problems with eighteen EPMM indicators were mentioned in forty-seven instances. The identified indicators were grouped into three domains: maternal health policy, financing, and service delivery. The final selection of indicators was made with inputs from global advisors to ensure harmonization of efforts.

Table 2.  Ten EPMM indicators for refinement with full metadata

CSE – comprehensive sexuality education, DHS – Demographic and Health Surveys, EPMM – Ending Preventable Maternal Mortality, GGS – Generations and Gender Surveys, HIV – human immunodeficiency virus, HPV – human papilloma virus, MICS – Multiple Indicator Cluster Surveys, RMNCAH – Reproductive, Maternal, Newborn, Child, and Adolescent Health, SDG – Sustainable Development Goals, WHO – World Health Organization

Consultation process

The full metadata (standard indicator name, definition, numerator, denominator, calculation, disaggregation, and data sources) were presented with an overview of data generated from the indicator across geographies and time. Speakers shared perspectives on specific problems with each indicator. These presentations provided an introduction for focused technical work.

Participants engaged in structured discussion of each indicator facilitated by the author to reach consensus on the nature and locus of problems within the metadata, and concrete solutions to address problems identified. Consensus was achieved through plenary discussion documented in real time on a projected screen and agreed by voice vote. A set of recommendations for each indicator was formulated.Key recommendations

Problems identified fell into eleven categories. Problem distribution and frequency across all ten indicators is summarized in Table 3.

Table 3.  Summary of problems identified in ten EPMM indicators

MH – maternal health, EPMM – Ending Preventable Maternal Mortality

Consultation 1: Selected Maternal Health Policy Indicators

1) Legal status of abortion

Recommendations:

1. Implement directionality in the value of the indicator based on evidence demonstrating the association between legal grounds and outcomes of interest (eg, safety, access) to allow tracking.
2. Create a scoring hierarchy that progresses from most to least restrictive, using a color coding system.
3. Transform the criteria from national categorical responses (Yes/No) to capture responses disaggregated by sub-national geographies.
4. Develop signal functions for abortion based on guidelines for accessibility, availability, acceptability, and quality (AAAQ) of abortion services in the WHO Global Abortion Policy Database (GAPD) [23].
5. Add sub-measures to allow global comparisons of abortion legality as part of the enabling environment for maternal health:
   1. Evidence of elements of AAAQ as defined in the WHO GAPD guidelines
   2. Types of provider authorized to provide legal abortion
   3. Census of authorized abortion providers

2) Is there a national policy to ensure engagement of civil society organization (CSO) representatives in periodic review of national programs for reproductive maternal newborn child adolescent health (RMNCAH)?

Recommendations:

1. Adjust the indicator to measure engagement directly. However, country representatives endorsed monitoring the existence of a policy requiring CSO representation until direct measurement of effective engagement is feasible.
2. Define engagement of CSO representatives:
   1. Incorporate lessons from Ocloo & Matthews (2016) to operationalize the definition of engagement in a meaningful way [24].
   2. Define measures and data sources to evidence engagement, eg, registry of public comment periods, minutes reflecting participation in formulation of policy revisions needed, record of interventions, etc.
3. Categorize responses by types of CSOs engaged (eg, international non-governmental organizations [NGOs], local NGOs), rather than by review of some vs. all components of RMNCAH programs. Include the latter as a disaggregation factor.
   1. Highlight inclusion of CSOs that represent marginalized populations within the definition of CSOs in the survey instructions. Consider national adaptations designating specified marginalized groups to drive improvement in their representation in those countries.
4. Reaffirm the intent of this indicator is for national-level monitoring. Prioritize engagement of subnational and national CSOs in review of national RMNCAH programs.
   1. Specify optimal respondents in the survey instructions (i.e., the data source).
5. Develop a scoring system based on organizational maturity, eg, a five-point scale from nascent to mature, using operational definitions to be included in the indicator.
6. Define “periodic review” as “assessment of progress on indicators in the national RMNCAH strategy” and specify that it must be “participatory”.
   1. Define periodicity (should be defined by those leading the RMNCAH programs)
      1. Align with the RMNCAH WHO Policy Survey.
      2. Align with the related EPMM indicator, number 3), below.
7. Require documentation of the written policy, with evidence of implementation guidelines within the national strategic document, as the data source.
   1. Ensure that the source document is appended.
   2. Include reports/minutes of the periodic reviews.

3) Presence of a national set of indicators with targets and annual report to inform annual health sector reviews and other planning cycles

Recommendations:

1. Clarify the intended construct for measurement, eg:
   1. Governance structure adequate to guide planning and monitoring of health issues and responses
   2. Country ownership of selection of indicators and targets for national MNH monitoring
   3. Improvement of the mechanism for national MNH monitoring overall
2. Separate the current indicator into three related but separate measures:
   1. Existence of national set on indicators/targets
   2. Analysis of the data through generation of an annual report
   3. Evidence of meaningful use of the information
3. Specify the level of use:
   1. For global monitoring, limit the indicator to existence of a national set of indicators with targets.
   2. For national/subnational use, include data to measure active monitoring and use of the national set of indicators with targets.
4. Adjust the indicator definition to “Specification of a national set of MNH indicators with annual reporting of current estimates/values, available in the public domain”.
   1. Remove “and other planning cycles”.
5. Specify that “current values” should be reported and capture periodicity for each component indicator, as not all indicators are annual.
6. Specify a scoring mechanism, modeled after those proposed by SCORE or MEASURE Evaluation [25,26].

4) Presence of laws and regulations that guarantee women aged 15-49 access to sexual and reproductive health (SRH) care, information, and education

Recommendations:

1. Conduct a systematic review of empirical evidence and/or human rights entitlements to substantiate the construct validity for each component.
2. Remove age limits from the indicator definition; consider any age limits in place a restriction.
3. Disaggregate by state for countries with differing laws and regulations for component parts.
   1. Harmonize with SDG 17.18.1.: “Proportion of sustainable development indicators produced at the national level with full disaggregation when relevant to the target, in accordance with the Fundamental Principles of Official Statistics”.
4. Revise the scoring mechanism to address the following specific problems:
   1. All components are arbitrarily equally weighted, but their specificity varies greatly (eg, “maternity care” is included as a single component).
   2. It is impossible to distinguish between national- and state-level variations.
   3. Subtracting barriers from enablers to calculate the indicator score is sensitive to the number of barriers and enablers included.
   4. The total score is calculated based on individual components, not section scores (the mean of components within each section). Calculating the total score by taking the average of the individual components across all sections arbitrarily assigns more importance to sections with more components than others, rather than giving all four sections equal weight.

5) Proportion of women aged 15-49 who make their own informed decisions regarding sexual relations, contraceptive use, and reproductive health care

Recommendations:

1. Articulate the construct for measurement clearly, eg:
   1. Women’s bodily autonomy and agency over decisions that affect her personally
   2. Women’s empowerment within society and/or within her intimate partner relationships
2. Evaluate whether the intended construct encapsulates all three components of this indicator. Conduct validation research to ascertain whether data for all components demonstrate convergent validity.
3. If the evaluation suggests no strong unifying construct, uncouple the components and report them separately.
   1. Provide a human rights- and evidence-based analysis of the basis for each component through a systematic review of the literature.
   2. Conduct qualitative research to explore social determinants that influence or explain the outcomes of interest.
4. Add supplemental response options to explore root cause factors that limit or influence decision-making, eg, access to financial resources, required 3rd party authorization, etc.
5. Correlate, validate, and harmonize with the SWPER survey-based index for women’s empowerment [27], which uses DHS data on decision making to allow comparable measures across time and countries.
6. Adjust the denominator.
   1. If the components are uncoupled, there is no need for a common denominator.
   2. Remove “currently using contraception” from the denominator, particularly for the question about a woman’s ability to refuse sexual intercourse.
   3. Expand the survey and data sources beyond Demographic and Health Surveys (DHS) (eg, Multiple Indicator Cluster Surveys [MICs] or Performance Monitoring for Action [PMA2020]) to address issues identified with the denominator that are dictated by the DHS format.
7. Remove the word “informed” from the definition, or define it operationally for all three components.
8. Revise the scoring module:
   1. Report each domain score and the total. Score components separately for each of the three domains, and take the average for each domain (instead of multiplying, which gives a value that is too small and hard to interpret).
   2. Make scoring binary for each component as follows:
      1. For Questions 1 & 2, collapse and report “Mainly alone” or “Joint decision” (affirmative responses that count toward empowerment) vs. “Mainly husband” or “partner and Other/Specify” (responses that do not count toward empowerment)
      2. For Question 3, report “Yes” vs. “Depends/Not Sure”
   3. Study and explore systematic differences between those who answer in the affirmative for all three questions vs. those who do not.

Consultation 2: Selected Maternal Health Financing Indicators

6) Out-of-pocket expenditure as a percentage of total expenditure on health

Recommendations:

1. Address out-of-pocket expenditure on maternal health specifically:
   1. Specify standard disaggregation factors, including disaggregation by MNH similar to International Conference on Population and Development (ICPD) global survey [28]; India’s National Family Health Survey [29], PMA2020 [30], DHS, Service Provision Assessments (SPA).
   2. Alternatively, create a RMNCH module similar to DHS context-specific modules.
2. Specify data sources. Revise the DHS maternal health module to include questions related to out-of-pocket maternal health expenditure as a percentage of total household expenditure.
3. Advocate to WHO and national governments to make all data sources and full metadata, and not just the final reported indicator value, available in the public domain:
   1. Request a public-access data hub at the country government level.
   2. Report total government expenditure disaggregated by condition.
   3. Make metadata available to allow examination of line item expenditure.
4. Improve and standardize the methodology:
   1. Improve survey methodology by implementing standard recall period, optimal number of questions, questions grouped by type of expenditure, and probes to capture non-service related expenditure.
   2. Capture the estimated opportunity cost to people who cannot access care because of cost-prohibitions, to make the indicator “pro-poor.”
   3. Adjust the denominator to total household expenditure (not total health expenditure) to harmonize with SDG Target 3.8.2 [31], so this indicator is no longer constrained by National Health Account limitations.
   4. Disaggregate by funding source, using coding similar to the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) and World Health Organization (WHO), which allow disaggregation of donors.
5. Improve reporting:
   1. Report both directly-derived country values and data from special surveys not constrained by the national accounting framework (which requires a zero balance) separately, and triangulate to compare validity of these estimates.
   2. Regularly report the percentage of out-of-pocket expenditure attributable to maternal health in comparison to the percentage of out-of-pocket expenditure for other disease conditions (these data are available for many countries but are not routinely reported) [32].
6. Ensure intersectoral coordination between data custodians and stewards in the finance and health sectors at global and country levels:
   1. The data custodian for this indicator at the global level is the WHO National Health Accounts team in the Health Financing division, and at country level, the National Statistical Offices/National Health Accounts. At global level, ensure ongoing internal coordination with WHO divisions of Sexual and Reproductive Health (SRH) and Maternal Child Adolescent Health (MCA), and at country level with Ministries of Health maternal health divisions to improve this indicator for maternal health monitoring.

7) Are the following (maternal health-related) services provided free of charge at point of use in the public sector for women of reproductive age?

Recommendations:

1. Enumerate specific services in the area of childbirth to reflect lifesaving interventions for complications. Alternatively, define a minimum essential covered services package.
2. Change the estimation method to calculate this indicator by type of service that should be free rather than category of woman who must pay, for the following reasons:
   1. Some services are more likely to throw users into catastrophic spending (eg, C-section has greater costs incurred than immunization)
   2. This method still allows disaggregation by individual-level equity factors (wealth, age, geography, etc.)
   3. Evidence shows that targeting is less effective than universal coverage and has human rights implications.
3. Remove age limits.
4. Change the data source to use primary data collected via household or facility survey from women on any charges, formal or informal, that they have paid for care.

8) Costed implementation plan for maternal, newborn, and child health (MNCH)

Recommendations:

1. Clarify the underlying construct for measurement: national governance capacity to develop, cost, execute, and review a plan for MNCH.
2. Develop additional questions and analysis to strengthen the indicator’s ability to capture the intended construct:
   1. Start with the following categorical question: “Is there a stand-alone costed national plan for MNCH (that is not just part of a larger health strategy)?”
   2. Include further probes to determine the quality of the costing exercise (eg, does it include current/capital costs?)
   3. Given the trend toward decentralized health systems, measure the national government’s function to harmonize:
      1. across accounts
      2. costed plans from subnational level
      3. different financing sources (private sector, debt funding, donor funding)
3. Develop a tool to systematically assess the adequacy of the costing exercise and data sources submitted, to explore national costing capacity.
4. Expand the definition of a “national implementation plan” to include subnational plans, if these are the basis for planning and accounting.
   1. Add a discriminating question first, to determine whether the country is a federal state with decentralized planning (“Yes”/”No”).
   2. Measure the proportion of funding for the Consumer Price Index (CPI) that is budgeted at subnational level.
   3. Systematically analyze national governance in federal/decentralized states, as well as coordination of plans and budgets between the Ministries of Health and Finance. Collect evidence of effective coordination.
5. Evaluate the response rate and effectiveness of the survey questions through cognitive interviews, item analysis, etc. and implement changes to improve survey quality.

9) Annual reviews are conducted of health spending from all financial sources, including RMNCH spending, as part of broader health sector reviews

Recommendations:

1. Clarify the construct for measurement. Specify that the outcome of interest is occurrence of a routine “broad health sector review”, and the factor tracked is whether it includes review of health spending from all sources by condition (including RMNCH).
2. Define “broad health sector review” and specify the inputs that should be included for review.
3. Determine the optimal frequency for the review, given the burden and the periodicity for updates to the data that are included.
4. Specify that “all financial sources” include both government and external sources.
5. Adjust the question as follows:
   • “Is there a national health sector review?” (Yes/No)
     1. If Yes, “How often? When was the last one?”
     2. “Does it include review of health spending? If so, from which sources (enumerate financial sources that should be included)?”
     3. “Does it review spending by condition? If so, does that include RMNCH?”
     4. Specify that documents must be appended.
6. Identify a data custodian for this indicator, eg,:
   1. WHO RMNCAH Policy Survey
   2. Partnership for MNCH (PMNCH) Health Financing Group
   3. Consult the Independent Accountability Panel, as this indicator was adapted from a recommendation made by its predecessor, (Commission on Information and Accountability) CoIA.

10) Percentage of total health expenditure spent on reproductive, maternal, newborn, and child health

Recommendations:

1. To emphasize this indicator’s focus on accountability:
   1. Revise the numerator to focus on government sources only. (Note: this will exclude the majority of the budget, which comes from ODA, in many countries). Alternatively, disaggregate by source.
   2. Adjust the denominator to government expenditure instead of all sources.
2. Report absolute expenditure by condition rather than the percentage of total expenditure. A relative measure risks pitting conditions against each other.
3. Disaggregate government spending versus ODA/other spending to help push governments toward self-sufficiency in the area of health where there is disproportionate reliance on ODA.
4. Demand transparency of data sources (national budgets), to allow CSOs and other stakeholders to review and calculate the disaggregated data on spending by condition.
5. Conduct validation research to explore the relative validity of similar indicators measuring this construct, e.g. “Current country health expenditure per capita (including specifically on RMNCAH) financed from domestic sources” [33]. Harmonize the indicator reported by global initiatives working to improve tracking of ODA and domestic health financing based on the results.
6. Identify an appropriate data custodian for this indicator, eg;
   1. GFF
   2. WHO Global Action Plan and partners [34]
   3. CSO budget accountability organizations [35,36]

DISCUSSION AND CONCLUSIONS

This paper summarizes weaknesses encountered with ten global maternal health indicators prioritized for monitoring progress toward ending preventable maternal mortality and proposes specific solutions to strengthen them. Eleven types of problems were identified, about which some generalizations can be made. The recommended solutions are, for the most part, specific to each indicator.

Of note, lack of clarity and conceptual precision in the underlying construct for measurement was identified in all ten indicators and, thus, construct validity was suboptimal for all indicators reviewed. Similarly, a majority of indicators exhibited issues with components in the numerator or denominator, and lacked operationalized definitions for key terms. Benova and colleagues [37] highlight the primary importance of theoretical clarity about the concept intended for measurement, including its intended purpose, meaningfulness, and utility, in their scoping review and definitional framework of indicator validity. Construct validity is overarching and subsumes other types of validity, since accurate measurement of a poorly operationalized or irrelevant concept will still lack validity. Furthermore, poor operationalization of the construct into the components of the numerator and denominator, or of specific terms therein, are further threats to validity.

For a majority of indicators, data sources were not standard, not validated, or not available in the public domain. These findings underscore calls for greater data transparency to build trust in global health measures [38], and in fiscal governance for health [39] by scholars and advocates who highlight that indicator data sources should be available in the public domain to allow stakeholders to replicate, verify, and improve indicators of importance in their context. In maternal health financing, lack of transparent data are further compounded by lack of disaggregation by condition, making it especially difficult to track adequate budget allocation, actual spending, and out-of-pocket expenditure on maternal health specifically.

A problem intrinsic to many maternal health policy indicators is that they document the presence of a policy rather than its performance upon implementation. Without defined targets, values, directionality, or a scoring mechanism to measure trends, it is difficult to use them to track change. Issues with the methods for estimation were identified in 4/5 of maternal health policy indicators and 3/5 of finance indicators.

Our consultation process produced concrete recommendations to strengthen indicators identified as among the best available measures for tracking progress toward priority recommendations in the EPMM Strategies. A strength of this process was that it included representatives of the data custodian agencies for indicators under discussion, as well as academic and programmatic experts from a range of country contexts. A limitation is that expert opinion from non-systematic review is a relatively weak level of evidence [40]. However, these recommendations are grounded in a high level of collective experience and specific expertise from diverse sources.

These recommendations if implemented can improve the construct validity, reliability, and data quality of important indicators for ending preventable maternal mortality. Further validation research is needed for many maternal health indicators in use, especially at health policy and governance levels, including those covered in this review.

Additional material

Online Supplementary Document

Acknowledgements

We gratefully acknowledge the following people for their assistance in planning and executing these consultations or contributing to the development of the paper: Winfred Dotse-Gborgbortsi; Tiziana Leone; Zoe Matthews; Jean-Pierre Monet; Allisyn Moran; Jennifer Requejo; Marta Schaaf; Malia Skjefte; Sarah Smiley; Joe Strong; Kimberly Whipkey. We thank all participants in the IMHM Indicator Refinement Consultations for contributing their expertise and opinions. The list of participants and presenters is included as a supplemental file, and presentations are available from the corresponding author upon reasonable request.

Availability of data and materials: All data generated or analyzed during this study are included in this published article (and its supplementary information files), or are available at: https://doi.org/10.7910/DVN/BFQNLY. This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.