Impact factor (WEB OF SCIENCE - Clarivate)

2 year: 7.664 | 5 year: 7.127

COVID-19Viewpoints

Validity issues in measures of COVID-19 preventive behaviours

Junhua Dang1,2, Lile Jia3,4

1 School of Psychology, Beijing Sport University, Beijing, China
2 Department of Surgical Sciences, Uppsala University, Uppsala, Sweden
3 Department of Psychology, National University of Singapore, Singapore
4 Center for Population Health, National University of Singapore, Singapore

DOI: 10.7189/jogh.13.03026

Share:

Facebook
Twitter
LinkedIn
Print Friendly, PDF & Email

The global coronavirus disease 2019 (COVID-19) outbreak has become the most severe crisis in recent history. The virus spreads primarily through droplets people send out when they talk, sneeze, or cough. Therefore, preventive behaviours such as mask-wearing, social distancing, and movement restraint are integral to public health campaigns against the COVID-19 pandemic. Accordingly, scientists have investigated community- or individual-level predictors of preventive behaviours through either indirect or direct measuring approaches. Indirectly, researchers used morbidity/mortality as an inverse proxy for preventive behaviours, assuming low morbidity/mortality as an outcome of more preventive behaviours. Directly, researchers either collected people’s self-reports of preventive behaviours or employed objective measures such as mobility trends recorded by mobile phones.

No measurement, however, is perfect. Here we discuss the validity issues in the three measurement types by scrutinising the extent to which they represent what they are supposed to capture (i.e. preventive behaviours) and provide recommendations on how to best utilise these measures (Table 1).

Table 1.  Comparing the three measurement types of preventive behaviours

WordPress Data Table

VALIDITY ISSUES

First, officially reported morbidity/mortality is highly inaccurate. For morbidity, the difficulty in detecting asymptomatic infections and limitations due to varying testing capacity between countries has likely resulted in substantial underreporting of cases. For instance, it was estimated that only 1.4% of infections in Africa were reported [1]. Mortality data do not fare better. Based on data from large-scale surveys, excess deaths, burials, cemeteries, crematoriums, and hospital mortuaries, recent studies estimated that officially reported COVID-19 deaths represented only a portion of actual deaths [2], ranging from 75% in Europe [3] to only 12.5% in India [4]. The issue with reporting inaccuracy is not unique to COVID-19 but is common in most infectious disease outbreaks. For instance, the estimated deaths caused by malaria worldwide were four times higher than official statistics [5]. A retrospective analysis based on records of burials during the 2014-2016 Ebola epidemic in Sierra Leone revealed that the number of estimated deaths deviated significantly from that of reported cases, especially in regions far from the capital district [6]. This problem has not received adequate scholarly attention. The wide variation in inaccuracy prevents any cross-country comparisons. One study, for instance, found that, during the COVID-19 pandemic, regions with tight cultures and countries with stricter rules and punishments for deviance (e.g. Singapore and South Korea), had fewer reported infections and deaths than regions with loose cultures and countries that have weaker norms and were more permissive for atypical behaviour (e.g. Brazil and Spain) [7]. While the researchers concluded that the strong social norms in tight cultures encouraged collective, preventive behaviours, the results could also have come from the cross-country inaccuracies in morbidity/mortality.

Second, self-reports suffer severely from social desirability bias, as respondents tend to report an inflated degree of adherence to preventive behaviours. For instance, in a large-scale worldwide survey (n = 49 968 across 67 countries), most participants reported complete compliance with all possible preventions. The global means of following guidelines for social distancing and physical hygiene were improbably high, at 8.60 and 8.21, respectively (scale range = 0 (strongly disagree) to 10 (strongly agree)) [8]. Similar patterns have also been found during the 2014-2016 Ebola epidemic. Almost all participants indicated that they washed hands nearly every time (mean = 4.51; scale range = 1 (never) to 5 (every time)) [9]. Statistically, the highly skewed data restricts the variability in people’s responses, greatly reducing statistical power. Conceptually, because people report their behaviours in a socially desirable manner (i.e. to be perceived as a good citizen), self-report data could substantially distort the true relationship between relevant factors and preventive behaviours.

Finally, although objective measures are more accurate than morbidity/mortality and self-reports, whether they can represent preventive behaviours is unclear. For example, researchers consider high mobility as evidence of noncompliance with preventive guidelines. However, people may move a lot to conduct daily activities (e.g. grocery shopping) while strictly complying with relevant rules (e.g. mask-wearing, two-meter social distancing). Therefore, such data must be interpreted with caution.

RECOMMENDATIONS

Morbidity/mortality should only be used to compare communities covered by the same political and health system. Though inaccuracy may still exist, the bias in reporting should be unidirectional and otherwise similar, allowing for the association between community-level predictors and preventive behaviours to be estimated. Further, researchers should consider excess mortality a better indicator than officially reported deaths [2]; this variable is now tracked by the World Mortality Data set [10].

Self-reports should only be used with large sample sizes, which helps alleviate statistical concerns (i.e. restricted variability). Conceptual concerns (i.e. indicative of impression management rather than preventive behaviours) are more problematic. It is best to cross-check self-reports and objective measures for convergent evidence. For example, collectivism has been associated with both self-reports of obeying shelter-in-place orders and mobility trends indicative of staying-at-home duration [11]. In such cases, we can be more confident of our conclusions despite the validity issues in both measures.

Objective measures’ validity may depend on such contextual factors as policy stringency. For example, during the full lockdown period, mobility trends public spaces are a good indicator of non-compliance with preventive guidelines. However, when the lockdown is only partially or loosely implemented, mobility trends alone can be seriously biased without corroborating evidence (e.g. images or videos). Unfortunately, objective data reflecting other preventive behaviours like mask-wearing and physical hygiene are less available. Therefore, mobility data need more nuanced analysis and data from other sources need to be further developed.

CONCLUSION

Research on preventive behaviours during the COVID-19 pandemic showcases the importance of behavioural science. However, scholars’ enthusiasm for uncovering societally impactful findings must be matched with earnest scrutiny of their methodology. We call for some healthy reflection in our work.

[1] Funding: Junhua Dang was supported by the Swedish Research Council (2018-06664) and National Nature Science Foundation of China (Project 31871098). Lile Jia was supported by Grant E-101-00-0002-01 from the National University of Singapore.

[2] Authorship contributions: Junhua Dang developed the research idea and both authors contributed equally to the writing and revising.

[3] Disclosure of interest: The authors completed the ICMJE Disclosure of Interest Form (available upon request from the corresponding author) and disclose no relevant interests.

references

[1] JW Cabore, HC Karamagi, HK Kipruto, JK Mungatu, JA Asamani, and B Droti. COVID-19 in the 47 countries of the WHO African region: a modelling analysis of past trends and future patterns. Lancet Glob Health. 2022;10:e1099-114. DOI: 10.1016/S2214-109X(22)00233-9. [PMID:35659911]

[2] C Whittaker, PGT Walker, M Alhaffar, A Hamlet, BA Djaafara, and A Ghani. Under-reporting of deaths limits our understanding of true burden of covid-19. BMJ. 2021;375:n2239 DOI: 10.1136/bmj.n2239. [PMID:34642172]

[3] LS Vestergaard, J Nielsen, L Richter, D Schmid, N Bustos, and T Braeye. Excess all-cause mortality during the COVID-19 pandemic in Europe – preliminary pooled estimates from the EuroMOMO network, March to April 2020. Euro Surveill. 2020;25:2001214. DOI: 10.2807/1560-7917.ES.2020.25.26.2001214. [PMID:32643601]

[4] P Jha, Y Deshmukh, C Tumbe, W Suraweera, A Bhowmick, and S Sharma. COVID mortality in India: National survey data and health facility deaths. Science. 2022;375:667-71. DOI: 10.1126/science.abm5154. [PMID:34990216]

[5] World Health Organization. World Malaria Report 2019. 2019. Available: www.who.int/malaria. Accessed: 20 April 2023.

[6] BD Dalziel, MSY Lau, A Tiffany, A McClelland, J Zelner, and JR Bliss. Unreported cases in the 2014-2016 Ebola epidemic: Spatiotemporal variation, and implications for estimating transmission. PLoS Negl Trop Dis. 2018;12:e0006161. DOI: 10.1371/journal.pntd.0006161. [PMID:29357363]

[7] MJ Gelfand, JC Jackson, X Pan, D Nau, D Pieper, and E Denison. The relationship between cultural tightness–looseness and COVID-19 cases and deaths: a global analysis. Lancet Planet Health. 2021;5:e135-44. DOI: 10.1016/S2542-5196(20)30301-6. [PMID:33524310]

[8] JJ Van Bavel, A Cichocka, V Capraro, H Sjåstad, JB Nezlek, and T Pavlović. National identity predicts public health support during a global pandemic. Nat Commun. 2022;13:517 DOI: 10.1038/s41467-021-27668-9. [PMID:35082277]

[9] AE Gamma, J Slekiene, and HJ Mosler. The Impact of Various Promotional Activities on Ebola Prevention Behaviors and Psychosocial Factors Predicting Ebola Prevention Behaviors in the Gambia Evaluation of Ebola Prevention Promotions. Int J Environ Res Public Health. 2019;16:2020 DOI: 10.3390/ijerph16112020. [PMID:31174301]

[10] A Karlinsky and D Kobak. Tracking excess mortality across countries during the covid-19 pandemic with the world mortality dataset. eLife. 2021;10:e69336. DOI: 10.7554/eLife.69336. [PMID:34190045]

[11] Z Feng, K Zou, and K Savani. Cultural antecedents of virus transmission: Individualism is associated with lower compliance with social distancing rules during the COVID-19 pandemic. J Pers Soc Psychol. 2023;124:461-482. DOI: 10.1037/pspa0000322. [PMID:35980692]

Correspondence to:
Junhua Dang
Beijing Sport University
No. 48, information Road, Haidian District, Beijing
China
[email protected]
Lile Jia
National University of Singapore
21 Lower Kent Ridge Road, Singapore
Singapore
[email protected]