quantitative research articles health care

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Quantitative Results of a National Intervention to Prevent Hospital-Acquired Catheter-Associated Urinary Tract Infection: A Pre-Post Observational Study

Affiliations.

1 University of Michigan Medical School and Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan (J.M., M.T.G., S.S.).
2 University of Michigan School of Nursing, Ann Arbor, Michigan (M.M.).
3 University of Michigan Medical School, Ann Arbor, Michigan (J.M.A., A.S.).
4 Integrated Clinical Services Team, Trinity Health, Livonia, Michigan (R.N.O.).
5 Health Research & Educational Trust, American Hospital Association, Chicago, Illinois (A.J.R.).
6 Center for Clinical Management Research, Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan (D.R.).
PMID: 31569231
DOI: 10.7326/M18-3534

Background: Many hospitals struggle to prevent catheter-associated urinary tract infection (CAUTI).

Objective: To evaluate the effect of a multimodal initiative on CAUTI in hospitals with high burden of health care-associated infection (HAI).

Design: Prospective, national, nonrandomized, clustered, externally facilitated, pre-post observational quality improvement initiative, for 3 cohorts active between November 2016 and May 2018.

Setting: Acute care, long-term acute care, and critical access hospitals, including intensive care and non-intensive care wards.

Participants: Target hospitals had a high burden of Clostridioides difficile infection plus central line-associated bloodstream infection, CAUTI, or hospital-onset methicillin-resistant Staphylococcus aureus bloodstream infection, defined as cumulative attributable differences above the first tertile in the Targeted Assessment for Prevention (TAP) strategy. Some additional nonrecruited hospitals also joined.

Intervention: Multimodal intervention, including Practice Change Assessment tool to identify infection prevention and control (IPC) and HAI prevention gaps; Web-based, on-demand modules involving onboarding, foundational IPC practices, HAI-specific 2-tiered approach to prioritize and implement interventions, and TAP resources; monthly webinars; state partner-led in-person meetings; and feedback. State partners made site visits to at least 50% of their enrolled hospitals, to support self-assessments and coach.

Measurements: Rates of CAUTI and urinary catheter device utilization ratio.

Results: Of 387 participating hospitals from 23 states and the District of Columbia, 361 provided CAUTI data. Over the study period, the unadjusted CAUTI rate was low and relatively stable, decreasing slightly from 1.12 to 1.04 CAUTIs per 1000 catheter-days. Catheter utilization decreased from 21.46 to 19.83 catheter-days per 100 patient-days from the pre- to the postintervention period.

Limitations: The intervention period was brief, with no assessment of fidelity. Baseline CAUTI rates were low. Patient characteristics were not assessed.

Conclusion: This multimodal intervention yielded no substantial improvements in CAUTI or urinary catheter utilization.

Primary funding source: Centers for Disease Control and Prevention.

PubMed Disclaimer

Publication types

Search in MeSH

Related information

Linkout - more resources, full text sources.

Ovid Technologies, Inc.
Genetic Alliance
MedlinePlus Health Information

Miscellaneous

NCI CPTAC Assay Portal

Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Research article
Open access
Published: 03 February 2021

A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines

Ellesha A. Smith ORCID: orcid.org/0000-0002-4241-7205 1 ,
Nicola J. Cooper 1 ,
Alex J. Sutton 1 ,
Keith R. Abrams 1 &
Stephanie J. Hubbard 1

BMC Public Health volume 21 , Article number: 278 ( 2021 ) Cite this article

4209 Accesses

5 Citations

3 Altmetric

Metrics details

The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development (including meta-analysis) for dealing with heterogeneity of intervention effects, inappropriate ‘lumping’ of interventions, adjusting for different populations and outcomes and the inclusion of various study types. Growing awareness of the importance of using all available evidence has led to the publication of guidance documents for implementing methods to improve decision making by answering policy relevant questions.

The first part of this paper reviews the methods used to synthesise quantitative effectiveness evidence in public health guidelines by the National Institute for Health and Care Excellence (NICE) that had been published or updated since the previous review in 2012 until the 19th August 2019.The second part of this paper provides an update of the statistical methods and explains how they address issues related to evaluating effectiveness evidence of public health interventions.

The proportion of NICE public health guidelines that used a meta-analysis as part of the synthesis of effectiveness evidence has increased since the previous review in 2012 from 23% (9 out of 39) to 31% (14 out of 45). The proportion of NICE guidelines that synthesised the evidence using only a narrative review decreased from 74% (29 out of 39) to 60% (27 out of 45).An application in the prevention of accidents in children at home illustrated how the choice of synthesis methods can enable more informed decision making by defining and estimating the effectiveness of more distinct interventions, including combinations of intervention components, and identifying subgroups in which interventions are most effective.

Conclusions

Despite methodology development and the publication of guidance documents to address issues in public health intervention evaluation since the original review, NICE public health guidelines are not making full use of meta-analysis and other tools that would provide decision makers with fuller information with which to develop policy. There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making.

Peer Review reports

To make well-informed decisions and provide the best guidance in health care policy, it is essential to have a clear framework for synthesising good quality evidence on the effectiveness and cost-effectiveness of health interventions. There is a broad range of methods available for evidence synthesis. Narrative reviews provide a qualitative summary of the effectiveness of the interventions. Meta-analysis is a statistical method that pools evidence from multiple independent sources [ 1 ]. Meta-analysis and more complex variations of meta-analysis have been extensively applied in the appraisals of clinical interventions and treatments, such as drugs, as the interventions and populations are clearly defined and tested in randomised, controlled conditions. In comparison, public health studies are often more complex in design, making synthesis more challenging [ 2 ].

Many challenges are faced in the synthesis of public health interventions. There is often increased methodological heterogeneity due to the inclusion of different study designs. Interventions are often poorly described in the literature which may result in variation within the intervention groups. There can be a wide range of outcomes, whose definitions are not consistent across studies. Intermediate, or surrogate, outcomes are often used in studies evaluating public health interventions [ 3 ]. In addition to these challenges, public health interventions are often also complex meaning that they are made up of multiple, interacting components [ 4 ]. Recent guidance documents have focused on the synthesis of complex interventions [ 2 , 5 , 6 ]. The National Institute for Health and Care Excellence (NICE) guidance manual provides recommendations across all topics that are covered by NICE and there is currently no guidance that focuses specifically on the public health context.

Research questions

A methodological review of NICE public health intervention guidelines by Achana et al. (2014) found that meta-analysis methods were not being used [ 3 ]. The first part of this paper aims to update and compare, to the original review, the meta-analysis methods being used in evidence synthesis of public health intervention appraisals.

The second part of this paper aims to illustrate what methods are available to address the challenges of public health intervention evidence synthesis. Synthesis methods that go beyond a pairwise meta-analysis are illustrated through the application to a case study in public health and are discussed to understand how evidence synthesis methods can enable more informed decision making.

The third part of this paper presents software, guidance documents and web tools for methods that aim to make appropriate evidence synthesis of public health interventions more accessible. Recommendations for future research and guidance production that can improve the uptake of these methods in a public health context are discussed.

Update of NICE public health intervention guidelines review

Nice guidelines.

The National Institute for Health and Care Excellence (NICE) was established in 1999 as a health authority to provide guidance on new medical technologies to the NHS in England and Wales [ 7 ]. Using an evidence-based approach, it provides recommendations based on effectiveness and cost-effectiveness to ensure an open and transparent process of allocating NHS resources [ 8 ]. The remit for NICE guideline production was extended to public health in April 2005 and the first recommendations were published in March 2006. NICE published ‘Developing NICE guidelines: the manual’ in 2006, which has been updated since, with the most recent in 2018 [ 9 ]. It was intended to be a guidance document to aid in the production of NICE guidelines across all NICE topics. In terms of synthesising quantitative evidence, the NICE recommendations state: ‘meta-analysis may be appropriate if treatment estimates of the same outcome from more than 1 study are available’ and ‘when multiple competing options are being appraised, a network meta-analysis should be considered’. The implementation of network meta-analysis (NMA), which is described later, as a recommendation from NICE was introduced into the guidance document in 2014, with a further update in 2018.

Background to the previous review

The paper by Achana et al. (2014) explored the use of evidence synthesis methodology in NICE public health intervention guidelines published between 2006 and 2012 [ 3 ]. The authors conducted a systematic review of the methods used to synthesise quantitative effectiveness evidence within NICE public health guidelines. They found that only 23% of NICE public health guidelines used pairwise meta-analysis as part of the effectiveness review and the remainder used a narrative summary or no synthesis of evidence at all. The authors argued that despite significant advances in the methodology of evidence synthesis, the uptake of methods in public health intervention evaluation is lower than other fields, including clinical treatment evaluation. The paper concluded that more sophisticated methods in evidence synthesis should be considered to aid in decision making in the public health context [ 3 ].

The search strategy used in this paper was equivalent to that in the previous paper by Achana et al. (2014)[ 3 ]. The search was conducted through the NICE website ( https://www.nice.org.uk/guidance ) by searching the ‘Guidance and Advice List’ and filtering by ‘Public Health Guidelines’ [ 10 ]. The search criteria included all guidance documents that had been published from inception (March 2006) until the 19th August 2019. Since the original review, many of the guidelines had been updated with new documents or merged. Guidelines that remained unchanged since the previous review in 2012 were excluded and used for comparison.

The guidelines contained multiple documents that were assessed for relevance. A systematic review is a separate synthesis within a guideline that systematically collates all evidence on a specific research question of interest in the literature. Systematic reviews of quantitative effectiveness, cost-effectiveness evidence and decision modelling reports were all included as relevant. Qualitative reviews, field reports, expert opinions, surveillance reports, review decisions and other supporting documents were excluded at the search stage.

Within the reports, data was extracted on the types of review (narrative summary, pairwise meta-analysis, network meta-analysis (NMA), cost-effectiveness review or decision model), design of included primary studies (randomised controlled trials or non-randomised studies, intermediate or final outcomes, description of outcomes, outcome measure statistic), details of the synthesis methods used in the effectiveness evaluation (type of synthesis, fixed or random effects model, study quality assessment, publication bias assessment, presentation of results, software). Further details of the interventions were also recorded, including whether multiple interventions were lumped together for a pairwise comparison, whether interventions were complex (made up of multiple components) and details of the components. The reports were also assessed for potential use of complex intervention evidence synthesis methodology, meaning that the interventions that were evaluated in the review were made up of components that could potentially be synthesised using an NMA or a component NMA [ 11 ]. Where meta-analysis was not used to synthesis effectiveness evidence, the reasons for this was also recorded.

Search results and types of reviews

There were 67 NICE public health guidelines available on the NICE website. A summary flow diagram describing the literature identification process and the list of guidelines and their reference codes are provided in Additional files 1 and 2 . Since the previous review, 22 guidelines had not been updated. The results from the previous review were used for comparison to the 45 guidelines that were either newly published or updated.

The guidelines consisted of 508 documents that were assessed for relevance. Table 1 shows which types of relevant documents were available in each of the 45 guidelines. The median number of relevant articles per guideline was 3 (minimum = 0, maximum = 10). Two (4%) of the NICE public health guidelines did not report any type of systematic review, cost-effectiveness review or decision model (NG68, NG64) that met the inclusion criteria. 167 documents from 43 NICE public health guidelines were systematic reviews of quantitative effectiveness, cost-effectiveness or decision model reports and met the inclusion criteria.

Narrative reviews of effectiveness were implemented in 41 (91%) of the NICE PH guidelines. 14 (31%) contained a review that used meta-analysis to synthesise the evidence. Only one (1%) NICE guideline contained a review that implemented NMA to synthesise the effectiveness of multiple interventions; this was the same guideline that used NMA in the original review and had been updated. 33 (73%) guidelines contained cost-effectiveness reviews and 34 (76%) developed a decision model.

Comparison of review types to original review

Table 2 compares the results of the update to the original review and shows that the types of reviews and evidence synthesis methodologies remain largely unchanged since 2012. The proportion of guidelines that only contain narrative reviews to synthesise effectiveness or cost-effectiveness evidence has reduced from 74% to 60% and the proportion that included a meta-analysis has increased from 23% to 31%. The proportion of guidelines with reviews that only included evidence from randomised controlled trials and assessed the quality of individual studies remained similar to the original review.

Characteristics of guidelines using meta-analytic methods

Table 3 details the characteristics of the meta-analytic methods implemented in 24 reviews of the 14 guidelines that included one. All of the reviews reported an assessment of study quality, 12 (50%) reviews included only data from randomised controlled trials, 4 (17%) reviews used intermediate outcomes (e.g. uptake of chlamydia screening rather than prevention of chlamydia (PH3)), compared to the 20 (83%) reviews that used final outcomes (e.g. smoking cessation rather than uptake of a smoking cessation programme (NG92)). 2 (8%) reviews only used a fixed effect meta-analysis, 19 (79%) reviews used a random effects meta-analysis and 3 (13%) did not report which they had used.

An evaluation of the intervention information reported in the reviews concluded that 12 (50%) reviews had lumped multiple (more than two) different interventions into a control versus intervention pairwise meta-analysis. Eleven (46%) of the reviews evaluated interventions that are made up of multiple components (e.g. interventions for preventing obesity in PH47 were made up of diet, physical activity and behavioural change components).

21 (88%) of the reviews presented the results of the meta-analysis in the form of a forest plot and 22 (92%) presented the results in the text of the report. 20 (83%) of the reviews used two or more forms of presentation for the results. Only three (13%) reviews assessed publication bias. The most common software to perform meta-analysis was RevMan in 14 (58%) of the reviews.

Reasons for not using meta-analytic methods

The 143 reviews of effectiveness and cost effectiveness that did not use meta-analysis methods to synthesise the quantitative effectiveness evidence were searched for reasons behind this decision. 70 reports (49%) did not give a reason for not synthesising the data using a meta-analysis and 164 reasons were reported which are displayed in Fig. 1 . Out of the remaining reviews, multiple reasons for not using a meta-analysis were given. 53 (37%) of the reviews reported at least one reason due to heterogeneity. 30 (21%) decision model reports did not give a reason and these are categorised separately. 5 (3%) reviews reported that meta-analysis was not applicable or feasible, 1 (1%) reported that they were following NICE guidelines and 5 (3%) reported that there were a lack of studies.

Frequency and proportions of reasons reported for not using statistical methods in quantitative evidence synthesis in NICE PH intervention reviews

The frequency of reviews and guidelines that used meta-analytic methods were plotted against year of publication, which is reported in Fig. 2 . This showed that the number of reviews that used meta-analysis were approximately constant but there is some suggestion that the number of meta-analyses used per guideline increased, particularly in 2018.

Number of meta-analyses in NICE PH guidelines by year. Guidelines that were published before 2012 had been updated since the previous review by Achana et al. (2014) [ 3 ]

Comparison of meta-analysis characteristics to original review

Table 4 compares the characteristics of the meta-analyses used in the evidence synthesis of NICE public health intervention guidelines to the original review by Achana et al. (2014) [ 3 ]. Overall, the characteristics in the updated review have not much changed from those in the original. These changes demonstrate that the use of meta-analysis in NICE guidelines has increased but remains low. Lumping of interventions still appears to be common in 50% of reviews. The implications of this are discussed in the next section.

Application of evidence synthesis methodology in a public health intervention: motivating example

Since the original review, evidence synthesis methods have been developed and can address some of the challenges of synthesising quantitative effectiveness evidence of public health interventions. Despite this, the previous section shows that the uptake of these methods is still low in NICE public health guidelines - usually limited to a pairwise meta-analysis.

It has been shown in the results above and elsewhere [ 12 ] that heterogeneity is a common reason for not synthesising the quantitative effectiveness evidence available from systematic reviews in public health. Statistical heterogeneity is the variation in the intervention effects between the individual studies. Heterogeneity is problematic in evidence synthesis as it leads to uncertainty in the pooled effect estimates in a meta-analysis which can make it difficult to interpret the pooled results and draw conclusions. Rather than exploring the source of the heterogeneity, often in public health intervention appraisals a random effects model is fitted which assumes that the study intervention effects are not equivalent but come from a common distribution [ 13 , 14 ]. Alternatively, as demonstrated in the review update, heterogeneity is used as a reason to not undertake any quantitative evidence synthesis at all.

Since the size of the intervention effects and the methodological variation in the studies will affect the impact of the heterogeneity on a meta-analysis, it is inappropriate to base the methodological approach of a review on the degree of heterogeneity, especially within public health intervention appraisal where heterogeneity seems inevitable. Ioannidis et al. (2008) argued that there are ‘almost always’ quantitative synthesis options that may offer some useful insights in the presence of heterogeneity, as long as the reviewers interpret the findings with respect to their limitations [ 12 ].

In this section current evidence synthesis methods are applied to a motivating example in public health. This aims to demonstrate that methods beyond pairwise meta-analysis can provide appropriate and pragmatic information to public health decision makers to enable more informed decision making.

Figure 3 summarises the narrative of this part of the paper and illustrates the methods that are discussed. The red boxes represent the challenges in synthesising quantitative effectiveness evidence and refers to the section within the paper for more detail. The blue boxes represent the methods that can be applied to investigate each challenge.

Summary of challenges that are faces in the evidence synthesis of public health interventions and methods that are discussed to overcome these challenges

Evaluating the effect of interventions for promoting the safe storage of cleaning products to prevent childhood poisoning accidents

To illustrate the methodological developments, a motivating example is used from the five year, NIHR funded, Keeping Children Safe Programme [ 15 ]. The project included a Cochrane systematic review that aimed to increase the use of safety equipment to prevent accidents at home in children under five years old. This application is intended to be illustrative of the benefits of new evidence synthesis methods since the previous review. It is not a complete, comprehensive analysis as it only uses a subset of the original dataset and therefore the results are not intended to be used for policy decision making. This example has been chosen as it demonstrates many of the issues in synthesising effectiveness evidence of public health interventions, including different study designs (randomised controlled trials, observational studies and cluster randomised trials), heterogeneity of populations or settings, incomplete individual participant data and complex interventions that contain multiple components.

This analysis will investigate the most effective promotional interventions for the outcome of ‘safe storage of cleaning products’ to prevent childhood poisoning accidents. There are 12 studies included in the dataset, with IPD available from nine of the studies. The covariate, single parent family, is included in the analysis to demonstrate the effect of being a single parent family on the outcome. In this example, all of the interventions are made up of one or more of the following components: education (Ed), free or low cost equipment (Eq), home safety inspection (HSI), and installation of safety equipment (In). A Bayesian approach using WinBUGS was used and therefore credible intervals (CrI) are presented with estimates of the effect sizes [ 16 ].

The original review paper by Achana et al. (2014) demonstrated pairwise meta-analysis and meta-regression using individual and cluster allocated trials, subgroup analyses, meta-regression using individual participant data (IPD) and summary aggregate data and NMA. This paper firstly applies NMA to the motivating example for context, followed by extensions to NMA.

Multiple interventions: lumping or splitting?

Often in public health there are multiple intervention options. However, interventions are often lumped together in a pairwise meta-analysis. Pairwise meta-analysis is a useful tool for two interventions or, alternatively in the presence of lumping interventions, for answering the research question: ‘are interventions in general better than a control or another group of interventions?’. However, when there are multiple interventions, this type of analysis is not appropriate for informing health care providers which intervention should be recommended to the public. ‘Lumping’ is becoming less frequent in other areas of evidence synthesis, such as for clinical interventions, as the use of sophisticated synthesis techniques, such as NMA, increases (Achana et al. 2014) but lumping is still common in public health.

NMA is an extension of the pairwise meta-analysis framework to more than two interventions. Multiple interventions that are lumped into a pairwise meta-analysis are likely to demonstrate high statistical heterogeneity. This does not mean that quantitative synthesis could not be undertaken but that a more appropriate method, NMA, should be implemented. Instead the statistical approach should be based on the research questions of the systematic review. For example, if the research question is ‘are any interventions effective for preventing obesity?’, it would be appropriate to perform a pairwise meta-analysis comparing every intervention in the literature to a control. However, if the research question is ‘which intervention is the most effective for preventing obesity?’, it would be more appropriate and informative to perform a network meta-analysis, which can compare multiple interventions simultaneously and identify the best one.

NMA is a useful statistical method in the context of public health intervention appraisal, where there are often multiple intervention options, as it estimates the relative effectiveness of three or more interventions simultaneously, even if direct study evidence is not available for all intervention comparisons. Using NMA can help to answer the research question ‘what is the effectiveness of each intervention compared to all other interventions in the network?’.

In the motivating example there are six intervention options. The effect of lumping interventions is shown in Fig. 4 , where different interventions in both the intervention and control arms are compared. There is overlap of intervention and control arms across studies and interpretation of the results of a pairwise meta-analysis comparing the effectiveness of the two groups of interventions would not be useful in deciding which intervention to recommend. In comparison, the network plot in Fig. 5 illustrates the evidence base of the prevention of childhood poisonings review comparing six interventions that promote the use of safety equipment in the home. Most of the studies use ‘usual care’ as a baseline and compare this to another intervention. There are also studies in the evidence base that compare pairs of the interventions, such as ‘Education and equipment’ to ‘Equipment’. The plot also demonstrates the absence of direct study evidence between many pairs of interventions, for which the associated treatment effects can be indirectly estimated using NMA.

Network plot to illustrate how pairwise meta-analysis groups the interventions in the motivating dataset. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

Network plot for the safe storage of cleaning products outcome. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

An NMA was fitted to the motivating example to compare the six interventions in the studies from the review. The results are reported in the ‘triangle table’ in Table 5 [ 17 ]. The top right half of the table shows the direct evidence between pairs of the interventions in the corresponding rows and columns by either pooling the studies as a pairwise meta-analysis or presenting the single study results if evidence is only available from a single study. The bottom left half of the table reports the results of the NMA. The gaps in the top right half of the table arise where no direct study evidence exists to compare the two interventions. For example, there is no direct study evidence comparing ‘Education’ (Ed) to ‘Education, equipment and home safety inspection’ (Ed+Eq+HSI). The NMA, however, can estimate this comparison through the direct study evidence as an odds ratio of 3.80 with a 95% credible interval of (1.16, 12.44). The results suggest that the odds of safely storing cleaning products in the Ed+Eq+HSI intervention group is 3.80 times the odds in the Ed group. The results demonstrate a key benefit of NMA that all intervention effects in a network can be estimated using indirect evidence, even if there is no direct study evidence for some pairwise comparisons. This is based on the consistency assumption (that estimates of intervention effects from direct and indirect evidence are consistent) which should be checked when performing an NMA. This is beyond the scope of this paper and details on this can be found elsewhere [ 18 ].

NMA can also be used to rank the interventions in terms of their effectiveness and estimate the probability that each intervention is likely to be the most effective. This can help to answer the research question ‘which intervention is the best?’ out of all of the interventions that have provided evidence in the network. The rankings and associated probabilities for the motivating example are presented in Table 6 . It can be seen that in this case the ‘education, equipment and home safety inspection’ (Ed+Eq+HSI) intervention is ranked first, with a 0.87 probability of being the best intervention. However, there is overlap of the 95% credible intervals of the median rankings. This overlap reflects the uncertainty in the intervention effect estimates and therefore it is important that the interpretation of these statistics clearly communicates this uncertainty to decision makers.

NMA has the potential to be extremely useful but is underutilised in the evidence synthesis of public health interventions. The ability to compare and rank multiple interventions in an area where there are often multiple intervention options is invaluable in decision making for identifying which intervention to recommend. NMA can also include further literature in the analysis, compared to a pairwise meta-analysis, by expanding the network to improve the uncertainty in the effectiveness estimates.

Statistical heterogeneity

When heterogeneity remains in the results of an NMA, it is useful to explore the reasons for this. Strategies for dealing with heterogeneity involve the inclusion of covariates in a meta-analysis or NMA to adjust for the differences in the covariates across studies [ 19 ]. Meta-regression is a statistical method developed from meta-analysis that includes covariates to potentially explain the between-study heterogeneity ‘with the aim of estimating treatment-covariate interactions’ (Saramago et al. 2012). NMA has been extended to network meta-regression which investigates the effect of trial characteristics on multiple intervention effects. Three ways have been suggested to include covariates in an NMA: single covariate effect, exchangeable covariate effects and independent covariate effects which are discussed in more detail in the NICE Technical Support Document 3 [ 14 ]. This method has the potential to assess the effect of study level covariates on the intervention effects, which is particularly relevant in public health due to the variation across studies.

The most widespread method of meta-regression uses study level data for the inclusion of covariates into meta-regression models. Study level covariate data is when the data from the studies are aggregated, e.g. the proportion of participants in a study that are from single parent families compared to dual parent families. The alternative to study level data is individual participant data (IPD), where the data are available and used as a covariate at the individual level e.g. the parental status of every individual in a study can be used as a covariate. Although IPD is considered to be the gold standard for meta-analysis, aggregated level data is much more commonly used as it is usually available and easily accessible from published research whereas IPD can be hard to obtain from study authors.

There are some limitations to network meta-regression. In our motivating example, using the single parent covariate in a meta-regression would estimate the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families. This interpretation is not as useful as the analysis that uses IPD, which would give the relative difference of the intervention effects in a single parent family compared to a dual parent family. The meta-regression using aggregated data would also be susceptible to ecological bias. Ecological bias is where the effect of the covariate is different at the study level compared to the individual level [ 14 ]. For example, if each study demonstrates a relationship between a covariate and the intervention but the covariate is similar across the studies, a meta-regression of the aggregate data would not demonstrate the effect that is observed within the studies [ 20 ].

Although meta-regression is a useful tool for investigating sources of heterogeneity in the data, caution should be taken when using the results of meta-regression to explain how covariates affect the intervention effects. Meta-regression should only be used to investigate study characteristics, such as the duration of intervention, which will not be susceptible to ecological bias and the interpretation of the results (the effect of intervention duration on intervention effectiveness) would be more meaningful for the development of public health interventions.

Since the covariate of interest in this motivating example is not a study characteristic, meta-regression of aggregated covariate data was not performed. Network meta-regression including IPD and aggregate level data was developed by Samarago et al. (2012) [ 21 ] to overcome the issues with aggregated data network meta-regression, which is discussed in the next section.

Tailored decision making to specific sub-groups

In public health it is important to identify which interventions are best for which people. There has been a recent move towards precision medicine. In the field of public health the ‘concept of precision prevention may [...] be valuable for efficiently targeting preventive strategies to the specific subsets of a population that will derive maximal benefit’ (Khoury and Evans, 2015). Tailoring interventions has the potential to reduce the effect of inequalities in social factors that are influencing the health of the population. Identifying which interventions should be targeted to which subgroups can also lead to better public health outcomes and help to allocate scarce NHS resources. Research interest, therefore, lies in identifying participant level covariate-intervention interactions.

IPD meta-analysis uses data at the individual level to overcome ecological bias. The interpretation of IPD meta-analysis is more relevant in the case of using participant characteristics as covariates since the interpretation of the covariate-intervention interaction is at the individual level rather than the study level. This means that it can answer the research question: ‘which interventions work best in subgroups of the population?’. IPD meta-analyses are considered to be the gold standard for evidence synthesis since it increases the power of the analysis to identify covariate-intervention interactions and it has the ability to reduce the effect of ecological bias compared to aggregated data alone. IPD meta-analysis can also help to overcome scarcity of data issues and has been shown to have higher power and reduce the uncertainty in the estimates compared to analysis including only summary aggregate data [ 22 ].

Despite the advantages of including IPD in a meta-analysis, in reality it is often very time consuming and difficult to collect IPD for all of the studies [ 21 ]. Although data sharing is becoming more common, it remains time consuming and difficult to collect IPD for all studies in a review. This results in IPD being underutilised in meta-analyses. As an intermediate solution, statistical methods have been developed, such as the NMA in Samarago et al. (2012), that incorporates both IPD and aggregate data. Methods that simultaneously include IPD and aggregate level data have been shown to reduce uncertainty in the effect estimates and minimise ecological bias [ 20 , 21 ]. A simulation study by Leahy et al. (2018) found that an increased proportion of IPD resulted in more accurate and precise NMA estimates [ 23 ].

An NMA including IPD, where it is available, was performed, based on the model presented in Samarago et al. (2012) [ 21 ]. The results in Table 7 demonstrates the detail that this type of analysis can provide to base decisions on. More relevant covariate-intervention interaction interpretations can be obtained, for example the regression coefficients for covariate-intervention interactions are the individual level covariate intervention interactions or the ‘within study interactions’ that are interpreted as the effect of being in a single parent family on the effectiveness of each of the interventions. For example, the effect of Ed+Eq compared to UC in a single parent family is 1.66 times the effect of Ed+Eq compared to UC in a dual parent family but this is not an important difference as the credible interval crosses 1. The regression coefficients for the study level covariate-intervention interactions or the ‘between study interactions’ can be interpreted as the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families.

Complex interventions

In many public health research settings the complex interventions are comprised of a number of components. An NMA can compare all of the interventions in a network as they are implemented in the original trials. However, NMA does not tell us which components of the complex intervention are attributable to this effect. It could be that particular components, or the interacting effect of multiple components, are driving the effectiveness and other components are not as effective. Often, trials have not directly compared every combination of components as there are so many component combination options, it would be inefficient and impractical. Component NMA was developed by Welton et al. (2009) to estimate the effect of each component of the complex interventions and combination of components in a network, in the absence of direct trial evidence and answers the question: ‘are interventions with a particular component or combination of components effective?’ [ 11 ]. For example, for the motivating example, in comparison to Fig. 5 , which demonstrates the interventions that an NMA can estimate effectiveness, Fig. 6 demonstrates all of the possible interventions of which the effectiveness can be estimated in a component NMA, given the components present in the network.

Network plot that illustrates how component network meta-analysis can estimate the effectiveness of intervention components and combinations of components, even when they are not included in the direct evidence. Notation UC: Usual care, Ed: Education, Eq: Equipment, Installation, Ed+Eq: Education and equipment, Ed+HSI: Education and home safety inspection, Ed+In: Education and installation, Eq+HSI: Equipment and home safety inspection, Eq+In: equipment and installation, HSI+In: Home safety inspection and installation, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq+HSI+In: Equipment, home safety inspection and installation, Ed+Eq+HSI+In: Education, equipment, home safety inspection and installation

The results of the analyses of the main effects, two way effects and full effects models are shown in Table 8 . The models, proposed in the original paper by Welton et al. (2009), increase in complexity as the assumptions regarding the component effects relax [ 24 ]. The main effects component NMA assumes that the components in the interventions each have separate, independent effects and intervention effects are the sum of the component effects. The two-way effects models assumes that there are interactions between pairs of the components, so the effects of the interventions are more than the sum of the effects. The full effects model assumes that all of the components and combinations of the components interact. Component NMA did not provide further insight into which components are likely to be the most effective since all of the 95% credible intervals were very wide and overlapped 1. There is a lot of uncertainty in the results, particularly in the 2-way and full effects models. A limitation of component NMA is that there are issues with uncertainty when data is scarce. However, the results demonstrate the potential of component NMA as a useful tool to gain better insights from the available dataset.

In practice, this method has rarely been used since its development [ 24 – 26 ]. It may be challenging to define the components in some areas of public health where many interventions have been studied. However, the use of meta-analysis for planning future studies is rarely discussed and component NMA would provide a useful tool for identifying new component combinations that may be more effective [ 27 ]. This type of analysis has the potential to prioritise future public health research, which is especially useful where there are multiple intervention options, and identify more effective interventions to recommend to the public.

Further methods / other outcomes

The analysis and methods described in this paper only cover a small subset of the methods that have been developed in meta-analysis in recent years. Methods that aim to assess the quality of evidence supporting a NMA and how to quantify how much the evidence could change due to potential biases or sampling variation before the recommendation changes have been developed [ 28 , 29 ]. Models adjusting for baseline risk have been developed to allow for different study populations to have different levels of underlying risk, by using the observed event rate in the control arm [ 30 , 31 ]. Multivariate methods can be used to compare the effect of multiple interventions on two or more outcomes simultaneously [ 32 ]. This area of methodological development is especially appealing within public health where studies assess a broad range of health effects and typically have multiple outcome measures. Multivariate methods offer benefits over univariate models by allowing the borrowing of information across outcomes and modelling the relationships between outcomes which can potentially reduce the uncertainty in the effect estimates [ 33 ]. Methods have also been developed to evaluate interventions with classes or different intervention intensities, known as hierarchical interventions [ 34 ]. These methods were not demonstrated in this paper but can also be useful tools for addressing challenges of appraising public health interventions, such as multiple and surrogate outcomes.

This paper only considered an example with a binary outcome. All of the methods described have also been adapted for other outcome measures. For example, the Technical Support Document 2 proposed a Bayesian generalised linear modelling framework to synthesise other outcome measures. More information and models for continuous and time-to-event data is available elsewhere [ 21 , 35 – 38 ].

Software and guidelines

In the previous section, meta-analytic methods that answer more policy relevant questions were demonstrated. However, as shown by the update to the review, methods such as these are still under-utilised. It is suspected from the NICE public health review that one of the reasons for the lack of uptake of methods in public health could be due to common software choices, such as RevMan, being limited in their flexibility for statistical methods.

Table 9 provides a list of software options and guidance documents that are more flexible than RevMan for implementing the statistical methods illustrated in the previous section to make these methods more accessible to researchers.

In this paper, the network plot in Figs. 5 and 6 were produced using the networkplot command from the mvmeta package [ 39 ] in Stata [ 61 ]. WinBUGS was used to fit the NMA in this paper by adapting the code in the book ‘Evidence Synthesis for Decision Making in Healthcare’ which also provides more detail on Bayesian methods and assessing convergence of Bayesian models [ 45 ]. The model for including IPD and summary aggregate data in an NMA was based on the code in the paper by Saramago et al. (2012). The component NMA in this paper was performed in WinBUGS through R2WinBUGS, [ 47 ] using the code in Welton et al. (2009) [ 11 ].

WinBUGS is a flexible tool for fitting complex models in a Bayesian framework. The NICE Decision Support Unit produced a series of Evidence Synthesis Technical Support Documents [ 46 ] that provide a comprehensive technical guide to methods for evidence synthesis and WinBUGS code is also provided for many of the models. Complex models can also be performed in a frequentist framework. Code and commands for many models are available in R and STATA (see Table 9 ).

The software, R2WinBUGS, was used in the analysis of the motivating example. Increasing numbers of researchers are using R and so packages that can be used to link the two softwares by calling BUGS models in R, packages such as R2WinBUGS, can improve the accessibility of Bayesian methods [ 47 ]. The new R package, BUGSnet, may also help to facilitate the accessibility and improve the reporting of Bayesian NMA [ 48 ]. Webtools have also been developed as a means of enabling researchers to undertake increasingly complex analyses [ 52 , 53 ]. Webtools provide a user-friendly interface to perform statistical analyses and often help in the reporting of the analyses by producing plots, including network plots and forest plots. These tools are very useful for researchers that have a good understanding of the statistical methods they want to implement as part of their review but are inexperienced in statistical software.

This paper has reviewed NICE public health intervention guidelines to identify the methods that are currently being used to synthesise effectiveness evidence to inform public health decision making. A previous review from 2012 was updated to see how method utilisation has changed. Methods have been developed since the previous review and these were applied to an example dataset to show how methods can answer more policy relevant questions. Resources and guidelines for implementing these methods were signposted to encourage uptake.

The review found that the proportion of NICE guidelines containing effectiveness evidence summarised using meta-analysis methods has increased since the original review, but remains low. The majority of the reviews presented only narrative summaries of the evidence - a similar result to the original review. In recent years, there has been an increased awareness of the need to improve decision making by using all of the available evidence. As a result, this has led to the development of new methods, easier application in standard statistical software packages, and guidance documents. Based on this, it would have been expected that their implementation would rise in recent years to reflect this, but the results of the review update showed no such increasing pattern.

A high proportion of NICE guideline reports did not provide a reason for not applying quantitative evidence synthesis methods. Possible explanations for this could be time or resource constraints, lack of statistical expertise, being unaware of the available methods or poor reporting. Reporting guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), should be updated to emphasise the importance of documenting reasons for not applying methods, as this can direct future research to improve uptake.

Where it was specified, the most common reported reason for not conducting a meta-analysis was heterogeneity. Often in public health, the data is heterogeneous due to the differences between studies in population, design, interventions or outcomes. A common misconception is that the presence of heterogeneity implies that it is not possible to pool the data. Meta-analytic methods can be used to investigate the sources of heterogeneity, as demonstrated in the NMA of the motivating example, and the use of IPD is recommended where possible to improve the precision of the results and reduce the effect of ecological bias. Although caution should be exercised in the interpretation of the results, quantitative synthesis methods provide a stronger basis for making decisions than narrative accounts because they explicitly quantify the heterogeneity and seek to explain it where possible.

The review also found that the most common software to perform the synthesis was RevMan. RevMan is very limited in its ability to perform advanced statistical analyses, beyond that of pairwise meta-analysis, which might explain the above findings. Standard software code is being developed to help make statistical methodology and application more accessible and guidance documents are becoming increasingly available.

The evaluation of public health interventions can be problematic due to the number and complexity of the interventions. NMA methods were applied to a real Cochrane public health review dataset. The methods that were demonstrated showed ways to address some of these issues, including the use of NMA for multiple interventions, the inclusion of covariates as both aggregated data and IPD to explain heterogeneity, and the extension to component network meta-analysis for guiding future research. These analyses illustrated how the choice of synthesis methods can enable more informed decision making by allowing more distinct interventions, and combinations of intervention components, to be defined and their effectiveness estimated. It also demonstrated the potential to target interventions to population subgroups where they are likely to be most effective. However, the application of component NMA to the motivating example has also demonstrated the issues around uncertainty if there are a limited number of studies observing the interventions and intervention components.

The application of methods to the motivating example demonstrated a key benefit of using statistical methods in a public health context compared to only presenting a narrative review – the methods provide a quantitative estimate of the effectiveness of the interventions. The uncertainty from the credible intervals can be used to demonstrate the lack of available evidence. In the context of decision making, having pooled estimates makes it much easier for decision makers to assess the effectiveness of the interventions or identify when more research is required. The posterior distribution of the pooled results from the evidence synthesis can also be incorporated into a comprehensive decision analytic model to determine cost-effectiveness [ 62 ]. Although narrative reviews are useful for describing the evidence base, the results are very difficult to summarise in a decision context.

Although heterogeneity seems to be inevitable within public health interventions due to their complex nature, this review has shown that it is still the main reported reason for not using statistical methods in evidence synthesis. This may be due to guidelines that were originally developed for clinical treatments that are tested in randomised conditions still being applied in public health settings. Guidelines for the choice of methods used in public health intervention appraisals could be updated to take into account the complexities and wide ranging areas in public health. Sophisticated methods may be more appropriate in some cases than simpler models for modelling multiple, complex interventions and their uncertainty, given the limitations are also fully reported [ 19 ]. Synthesis may not be appropriate if statistical heterogeneity remains after adjustment for possible explanatory covariates but details of exploratory analysis and reasons for not synthesising the data should be reported. Future research should focus on the application and dissemination of the advantages of using more advanced methods in public health, identifying circumstances where these methods are likely to be the most beneficial, and ways to make the methods more accessible, for example, the development of packages and web tools.

There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making. This review has shown that the uptake of statistical methods for evaluating the effectiveness of public health interventions is slow, despite advances in methods that address specific issues in public health intervention appraisal and the publication of guidance documents to complement their application.

Availability of data and materials

The dataset supporting the conclusions of this article is included within the article.

Abbreviations

National institute for health and care excellence

Network meta-analysis

Individual participant data

Home safety inspection

Installation

Credible interval

Preferred reporting items for systematic reviews and meta-analyses

Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011, p. 98. (Technical Support Document in Evidence Synthesis; TSD2).

Higgins JPT, López-López JA, Becker BJ, et al.Synthesising quantitative evidence in systematic reviews of complex health interventions. BMJ Global Health. 2019; 4(Suppl 1):e000858. https://doi.org/10.1136/bmjgh-2018-000858 .

Article PubMed PubMed Central Google Scholar

Achana F, Hubbard S, Sutton A, Kendrick D, Cooper N. An exploration of synthesis methods in public health evaluations of interventions concludes that the use of modern statistical methods would be beneficial. J Clin Epidemiol. 2014; 67(4):376–90.

Article PubMed Google Scholar

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new medical research council guidance. Int J Nurs Stud. 2013; 50(5):587–92.

Caldwell DM, Welton NJ. Approaches for synthesising complex mental health interventions in meta-analysis. Evidence-Based Mental Health. 2016; 19(1):16–21.

Melendez-Torres G, Bonell C, Thomas J. Emergent approaches to the meta-analysis of multiple heterogeneous complex interventions. BMC Med Res Methodol. 2015; 15(1):47.

Article CAS PubMed PubMed Central Google Scholar

NICE. NICE: Who We Are. https://www.nice.org.uk/about/who-we-are . Accessed 19 Sept 2019.

Kelly M, Morgan A, Ellis S, Younger T, Huntley J, Swann C. Evidence based public health: a review of the experience of the national institute of health and clinical excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010; 71(6):1056–62.

NICE. Developing NICE Guidelines: The Manual. https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview . Accessed 19 Sept 2019.

NICE. Public Health Guidance. https://www.nice.org.uk/guidance/published?type=ph . Accessed 19 Sept 2019.

Welton NJ, Caldwell D, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: psychological interventions in coronary heart disease. Am J Epidemiol. 2009; 169(9):1158–65.

Ioannidis JP, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008; 336(7658):1413–5.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539–58.

Article Google Scholar

Dias S, Sutton A, Welton N, Ades A. NICE DSU Technical Support Document 3: Heterogeneity: Subgroups, Meta-Regression, Bias and Bias-Adjustment: National Institute for Health and Clinical Excellence; 2011, p. 76.

Kendrick D, Ablewhite J, Achana F, et al.Keeping Children Safe: a multicentre programme of research to increase the evidence base for preventing unintentional injuries in the home in the under-fives. Southampton: NIHR Journals Library; 2017.

Google Scholar

Lunn DJ, Thomas A, Best N, et al.WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility. Stat Comput. 2000; 10:325–37. https://doi.org/10.1023/A:1008929526011 .

Dias S, Caldwell DM. Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed. 2019; 104(1):8–12. https://doi.org/10.1136/archdischild-2018-315224. http://arxiv.org/abs/https://fn.bmj.com/content/104/1/F8.full.pdf.

Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades A. NICE DSU Technical Support Document 4: Inconsistency in Networks of Evidence Based on Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011. (NICE DSU Technical Support Document in Evidence Synthesis; TSD4).

Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and technical challenges in network meta-analysis. Ann Intern Med. 2013; 159(2):130–7.

Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Res Synth Methods. 2010; 1(1):2–19.

Saramago P, Sutton AJ, Cooper NJ, Manca A. Mixed treatment comparisons using aggregate and individual participant level data. Stat Med. 2012; 31(28):3516–36.

Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol. 2002; 55(1):86–94.

Article CAS PubMed Google Scholar

Leahy J, O’Leary A, Afdhal N, Gray E, Milligan S, Wehmeyer MH, Walsh C. The impact of individual patient data in a network meta-analysis: an investigation into parameter estimation and model selection. Res Synth Methods. 2018; 9(3):441–69.

Freeman SC, Scott NW, Powell R, Johnston M, Sutton AJ, Cooper NJ. Component network meta-analysis identifies the most effective components of psychological preparation for adults undergoing surgery under general anesthesia. J Clin Epidemiol. 2018; 98:105–16.

Pompoli A, Furukawa TA, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychol Med. 2018; 48(12):1945–53.

Rücker G, Schmitz S, Schwarzer G. Component network meta-analysis compared to a matching method in a disconnected network: A case study. Biom J. 2020. https://doi.org/10.1002/bimj.201900339 .

Efthimiou O, Debray TP, van Valkenhoef G, Trelle S, Panayidou K, Moons KG, Reitsma JB, Shang A, Salanti G, Group GMR. GetReal in network meta-analysis: a review of the methodology. Res Synth Methods. 2016; 7(3):236–63.

Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS ONE. 2014; 9(7):99682.

Article CAS Google Scholar

Phillippo DM, Dias S, Welton NJ, Caldwell DM, Taske N, Ades A. Threshold analysis as an alternative to grade for assessing confidence in guideline recommendations based on network meta-analyses. Ann Intern Med. 2019; 170(8):538–46.

Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 5: Evidence Synthesis in the Baseline Natural History Model: National Institute for Health and Clinical Excellence; 2011, p. 29. (NICE DSU Technical Support Document in Evidence Synthesis; TSD5).

Achana FA, Cooper NJ, Dias S, Lu G, Rice SJ, Kendrick D, Sutton AJ. Extending methods for investigating the relationship between treatment effect and baseline risk from pairwise meta-analysis to network meta-analysis. Stat Med. 2013; 32(5):752–71.

Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, White IR. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ (Clinical research ed.) 2017; 358:j3932. https://doi.org/10.1136/bmj.j3932 .

Achana FA, Cooper NJ, Bujkiewicz S, Hubbard SJ, Kendrick D, Jones DR, Sutton AJ. Network meta-analysis of multiple outcome measures accounting for borrowing of information across outcomes. BMC Med Res Methodol. 2014; 14(1):92.

Owen RK, Tincello DG, Keith RA. Network meta-analysis: development of a three-level hierarchical modeling approach incorporating dose-related constraints. Value Health. 2015; 18(1):116–26.

Jansen JP. Network meta-analysis of individual and aggregate level data. Res Synth Methods. 2012; 3(2):177–90.

Donegan S, Williamson P, D’Alessandro U, Garner P, Smith CT. Combining individual patient data and aggregate data in mixed treatment comparison meta-analysis: individual patient data may be beneficial if only for a subset of trials. Stat Med. 2013; 32(6):914–30.

Saramago P, Chuang L-H, Soares MO. Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data. BMC Med Res Methodol. 2014; 14(1):105.

Thom HH, Capkun G, Cerulli A, Nixon RM, Howard LS. Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension. BMC Med Res Methodol. 2015; 15(1):34.

Gasparrini A, Armstrong B, Kenward MG. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012; 31(29):3821–39.

Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in stata. PLoS ONE. 2013; 8(10):76654.

Rücker G, Schwarzer G, Krahn U, König J. netmeta: Network meta-analysis with R. R package version 0.5-0. 2014. R package version 0.5-0. Availiable: http://CRAN.R-project.org/package=netmeta .

van Valkenhoef G, Kuiper J. gemtc: Network Meta-Analysis Using Bayesian Methods. R package version 0.8-2. 2016. Available online at: https://CRAN.R-project.org/package=gemtc .

Lin L, Zhang J, Hodges JS, Chu H. Performing arm-based network meta-analysis in R with the pcnetmeta package. J Stat Softw. 2017; 80(5):1–25. https://doi.org/10.18637/jss.v080.i05 .

Rücker G, Schwarzer G. Automated drawing of network plots in network meta-analysis. Res Synth Methods. 2016; 7(1):94–107.

Welton NJ, Sutton AJ, Cooper N, Abrams KR, Ades A. Evidence Synthesis for Decision Making in Healthcare, vol. 132. UK: Wiley; 2012.

Book Google Scholar

Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for decision making 1: introduction. Med Decis Making Int J Soc Med Decis Making. 2013; 33(5):597–606. https://doi.org/10.1177/0272989X13487604 .

Sturtz S, Ligges U, Gelman A. R2WinBUGS: a package for running WinBUGS from R. J Stat Softw. 2005; 12(3):1–16.

Béliveau A, Boyne DJ, Slater J, Brenner D, Arora P. Bugsnet: an r package to facilitate the conduct and reporting of bayesian network meta-analyses. BMC Med Res Methodol. 2019; 19(1):196.

Neupane B, Richer D, Bonner AJ, Kibret T, Beyene J. Network meta-analysis using R: a review of currently available automated packages. PLoS ONE. 2014; 9(12):115065.

White IR. Multivariate random-effects meta-analysis. Stata J. 2009; 9(1):40–56.

Chaimani A, Salanti G. Visualizing assumptions and results in network meta-analysis: the network graphs package. Stata J. 2015; 15(4):905–50.

Owen RK, Bradbury N, Xin Y, Cooper N, Sutton A. MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta. Res Synth Methods. 2019; 10(4):569–81. https://doi.org/10.1002/jrsm.1373 .

Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton AJ. Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Med Res Methodol. 2019; 19(1):81.

Nikolakopoulou A, Higgins JPT, Papakonstantinou T, Chaimani A, Del Giovane C, Egger M, Salanti G. CINeMA: An approach for assessing confidence in the results of a network meta-analysis. PLoS Med. 2020; 17(4):e1003082. https://doi.org/10.1371/journal.pmed.1003082 .

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36(3):1–48.

Freeman SC, Carpenter JR. Bayesian one-step ipd network meta-analysis of time-to-event data using royston-parmar models. Res Synth Methods. 2017; 8(4):451–64.

Riley RD, Lambert PC, Staessen JA, Wang J, Gueyffier F, Thijs L, Boutitie F. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med. 2008; 27(11):1870–93.

Debray TP, Moons KG, van Valkenhoef G, Efthimiou O, Hummel N, Groenwold RH, Reitsma JB, Group GMR. Get real in individual participant data (ipd) meta-analysis: a review of the methodology. Res Synth Methods. 2015; 6(4):293–309.

Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance on Their Use. PLoS Med. 2015; 12(7):e1001855. https://doi.org/10.1371/journal.pmed.1001855 .

Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the prisma-ipd statement. JAMA. 2015; 313(16):1657–65.

StataCorp. Stata Statistical Software: Release 16. College Station: StataCorp LLC; 2019.

Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: a bayesian approach. Health Econ. 2004; 13(3):203–26.

Download references

Acknowledgements

We would like to acknowledge Professor Denise Kendrick as the lead on the NIHR Keeping Children Safe at Home Programme that originally funded the collection of the evidence for the motivating example and some of the analyses illustrated in the paper.

ES is funded by a National Institute for Health Research (NIHR), Doctoral Research Fellow for this research project. This paper presents independent research funded by the National Institute for Health Research (NIHR). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author information

Authors and affiliations.

Department of Health Sciences, University of Leicester, Lancaster Road, Leicester, UK

Ellesha A. Smith, Nicola J. Cooper, Alex J. Sutton, Keith R. Abrams & Stephanie J. Hubbard

You can also search for this author in PubMed Google Scholar

Contributions

ES performed the review, analysed the data and wrote the paper. SH supervised the project. SH, KA, NC and AS provided substantial feedback on the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Ellesha A. Smith .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

KA is supported by Health Data Research (HDR) UK, the UK National Institute for Health Research (NIHR) Applied Research Collaboration East Midlands (ARC EM), and as a NIHR Senior Investigator Emeritus (NF-SI-0512-10159). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. KA has served as a paid consultant, providing unrelated methodological advice, to; Abbvie, Amaris, Allergan, Astellas, AstraZeneca, Boehringer Ingelheim, Bristol-Meyers Squibb, Creativ-Ceutical, GSK, ICON/Oxford Outcomes, Ipsen, Janssen, Eli Lilly, Merck, NICE, Novartis, NovoNordisk, Pfizer, PRMA, Roche and Takeda, and has received research funding from Association of the British Pharmaceutical Industry (ABPI), European Federation of Pharmaceutical Industries & Associations (EFPIA), Pfizer, Sanofi and Swiss Precision Diagnostics. He is a Partner and Director of Visible Analytics Limited, a healthcare consultancy company.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Key for the Nice public health guideline codes. Available in NICEGuidelinesKey.xlsx .

Additional file 2

NICE public health intervention guideline review flowchart for the inclusion and exclusion of documents. Available in Flowchart.JPG .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Smith, E.A., Cooper, N.J., Sutton, A.J. et al. A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines. BMC Public Health 21 , 278 (2021). https://doi.org/10.1186/s12889-021-10162-8

Download citation

Received : 22 September 2020

Accepted : 04 January 2021

Published : 03 February 2021

DOI : https://doi.org/10.1186/s12889-021-10162-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Meta-analysis
Systematic review
Public health
Decision making
Evidence synthesis

BMC Public Health

ISSN: 1471-2458

General enquiries: [email protected]

quantitative research articles health care

Open access
Published: 25 August 2022

A quantitative study of nurses perception to advance directive in selected private and public secondary healthcare facilities in Ibadan, Nigeria

Oluwaseyi Emiola Ojedoyin 1 &
Ayodele Samuel Jegede 1

BMC Medical Ethics volume 23 , Article number: 87 ( 2022 ) Cite this article

8902 Accesses

3 Altmetric

Metrics details

The study evaluated nurses’ perceptions on the benefits, drawbacks, and their roles in initiating and implementing advance directives (AD) at private and public secondary healthcare units.

The study adopted a cross-sectional, comparative-descriptive research design and was anchored on the structural functional theory. A total of 401 nurses (131 private and 270 public) were chosen on purpose. The data was collected between January and March 2018 among nurses at the selected hospitals. Analysis was done via SPSSv28.0.1.0.

Compared to nurses working in private healthcare facilities (72.5%), the majority of nurses at the public healthcare facilities (75.2%) indicated a more favorable opinion of AD’s benefits and (61.9%) felt they had a substantial involvement in the development and execution of AD than their private counterpart (56.5%). Similarly, 60.7% of nurses employed by the government agreed that AD has some disadvantages compared to those employed by the private sector (58.8%). Significantly, Christian nurses are 0.53 times less likely than Muslims to contest AD’s benefits; 0.78 times less likely than Muslim to disagree that AD has flaws; and 1.30 times more likely than Muslim nurses to deny they contributed to the development and execution of AD, though not significant.

Making decisions at the end-of-life can be challenging, thus AD should be supported across the board in the healthcare industry. Nurses should be trained on their role in developing and implementing AD, as well as on its advantages and how to deal with its challenges.

Peer Review reports

Introduction

Humans are born with the fundamental right to life. Due to this, many view death as undesirable, and even healthcare professionals avoid the discussion [ 1 , 2 ]. However, death is an inevitable, natural occurrence that all patients with life limiting illnesses should be prepared for in order to minimize distress at the end stage of life. Advance care planning (APC) is a method of communicating intentions that allows patients to let their loved ones and healthcare providers know in advance how they would like to be treated. One strategy in APC that aid readiness for future illness-related incapacitation, patients’ autonomy and dignity is advance directive (AD). AD is a written document or spoken declaration that enables competent people to make and document their healthcare decisions in advance [ 3 , 4 , 5 , 6 ]. Although patient’s “written directives” is a helpful tool for determining their preferences, tradition still dominates in most Africa countries. AD is yet to be legalised in Nigeria [ 7 ]. However, patients verbally expressed their preferences of care to healthcare professionals, and some even name individuals to make treatment decisions on their behalf when they are incapacitated [ 7 ]. These do not only promote patient participation in EOL discussion but also mitigate the paternalistic aspect of Nigeria’s healthcare system [ 8 , 9 ].

The Nigeria healthcare unit is divided into 3—primary, secondary and tertiary. Healthcare facilities at each unit can be privately owned or publicly owned. The difference between the two hospitals are found in their governance—the former are owned and run by an individual or group of individuals while the later are managed and funded by the government. The secondary healthcare facilities—which was the focus in this study manage advanced medical conditions [ 10 ] and it had been shown that, private hospitals are mostly used by Nigerians [ 11 ]. Nurses at these two facilities play significant roles in patients’ care. They provide medical, emotional, educational, patient-centered care and also serve as mediator between patients and doctors [ 12 , 13 ]. These put them in the best position to help in advance care planning—a procedure for communicating patients’ intentions [ 14 ]. Therefore, comparing the viewpoints of these nurses regarding AD will help to determine how end-of-life care is provided at this healthcare unit. There is a paucity of data on nurses’ perceptions of AD in Nigeria, and no study has described nurses’ perceptions at both private and public secondary healthcare facilities to the best of our knowledge. Previous researches focused on patient perception of AD and advocacy for AD inclusion in the country’s healthcare system [ 5 , 7 , 15 ]. This study therefore compared perceptions of nurses at the private and public secondary healthcare facilities on the advantages, roles and shortcoming of ADs in Ibadan, Oyo state, Nigeria.

Theoretical orientation

Structural–functional model.

A sociological theory known as functionalism views society as an organism of several elements (social institutions) that work together to maintain and reproduce the society [ 16 ]. These social institutions are typical means by which a society can attend to and satisfy both its social and individual needs. For instance, hospital is a social institution with many healthcare professionals collaborating to provide the best possible healthcare services to the community. Social institutions are also examined by functionalists in terms of the roles they played. Hence, to comprehend every part of society (e.g. doctor, nurse, teacher, AD, etc.) and how they affect social cohesion, reproduction, or the effective operation of a larger community, the functions of such institutions, beliefs, or ideologies are taken into considerations.

Merton however proposed that not all structure, custom, religion, ideology etc., serves positive purposes because they may serve both manifest and latent functions [ 17 ]. The latent functions are elements of behaviour or functions that are not openly declared, recognised, desired or intended. While the manifest functions are elements of conduct or functions that are conscious and purposefully [ 17 ]. Both the latent and manifest functions of AD was examined in the present study.

Research design

The study was a cross-sectional comparative-descriptive research design.

Participants

Nurses working in government-owned (public) and privately-owned secondary healthcare institutions as well as nursing students at the chosen hospitals participated in the survey.

Study location

The study was carried out in Ibadan, the Oyo state capital of Nigeria. Ibadan was deliberately chosen because it is Nigeria’s third-most populous city after Lagos and Kano, and because the region has historically had limited access to health care services [ 18 ]. Six out of eleven local government areas (LGAs) in Ibadan were chosen for this study—Ibadan Northeast, Ibadan Southwest, Ibadan Southeast, Ibadan North, and Egbeda. The high number of secondary health care facilities in these LGA coupled with the fact that no study on AD has been carried out among nurses in these locations were a deciding factors.

Sampling technique

A convenient non-probability sampling method was used to select nurses. This was employed due to the low staff strength, heavy workload and burnout on available staff. Five general hospitals and ten private secondary hospitals were included in the study—because of the high proportion of private secondary health facilities to public secondary health care facilities in the location and Nigeria as a whole [ 19 ]. A total of four hundred and one (401) nurses—270 nurses from public and 131 nurses from private hospitals—participated in the study.

Research instrument

Questionnaire was used to elicit information from respondents. Data was gathered in 2018 between January and March. The surveys were distributed to all nurses on-duty at their offices. A total of 430 survey was distributed out of which 401 was returned, making a 93% response rate. A total of 7% of the data was missing because several nurses worked night shifts, took the survey home, went on leave, and neglected to return the questionnaire.

The survey questions were developed after careful examination of literature from various countries [ 20 , 21 , 22 , 23 ]. Additionally, the opinions of three experts on prospective contents that required evaluation were sought. The questions’ ambiguity, relevance, clarity, and comprehensiveness were also evaluated. They assessed the questionnaire’s validity in terms of both face and content. The comments was examined, and the changes were added in the final survey. However, pilot survey was not conducted.

The questionnaire comprises two sections. The first section was on respondents’ socio-demographical characteristics. The second section was on perception and comprises 13 items—4 questions on benefits of AD, 5 questions on nurses’ roles in the initiation and implementation of AD, and 4 questions on shortcomings of AD. A 5-point likert scale was used to grade the responses of the participants ranging from strongly agree (5) to strongly disagree (0).

Data analysis

Data entry, cleaning, and analysis were performed using SPSS 28.0.1.0. Descriptive statistics was calculated for the socio-demographic and perception of nurses to AD. For questions on benefit of and nurses role in AD initiation and implementation, strongly agree and agree responses were merged to form correct perception to AD while, neutral, disagree and strongly disagree was merged as incorrect response. For questions on shortcomings of AD, strongly agree, agree and neutral responses were merged to form incorrect perception to AD while, disagree and strongly disagree was merged as correct response. The score for a correct response was two, while the score for an incorrect response was zero. The mean was calculated and response below the mean was considered as negative perceptions and those above or within the mean as positive perception.

On both the total benefits and drawbacks questions, 75% percentile (scoring three or more out of the four questions) was defined as positive perception, while 25% percentile (scored one out of the four questions) was labeled as negative view. The percentiles for the role of nurses in the initiation and implementation of AD were 60% (scoring 3 or more out of the 5 questions) and 40% (scored 2 or fewer out of the 5 questions). Differences between public and private nurses and nurses religion was examine using the odd ratios.

Reliability assessment of the questionnaire was conducted using Cronbach’s alpha coefficient based on Heden scale as cited in Peicus et al. [ 21 ] internal reliability assessment and recommendation. It stated that, a scale is reliable if the Cronbach’s alpha is > 5. The Cronbach alpha for the study is (0.62).

Ethical consideration

The Oyo State Research Ethics Review Committee, with reference number AD13/479/837, as well as administrative officers from each of the chosen hospitals and each participant, gave their approval before the data collection began.

Characteristic and representative of nurses in the study

The complete list of participants characteristics is shown in Table 1 below. The majority of respondents are women (88.9% public and 96.9% private). The majority (56.7%) of staff members at public hospitals hold diploma degrees, with one (0.4%) PhD degree holder. In contrast to the government hospitals, where 44.1% of participants had more than ten years of work experience, more than half (55.7%) of the private participants are within 1–5 years of work experience group. Predominant group are Yoruba (94%), Christians (79.3%) and more respondents from the public hospital (67.3%).

Distribution of nurses perception of benefits of advance directive

As shown in Table 2 below, most of the nurses agreed the AD is helpful when deciding how to treat patients (public-94.5% and private-93.1%); makes decision easier (public-88.2%, private-93.1%), minimize family conflict (public-85.9%, private-80.1%) and majority felt it reduced wasteful spending (public 77%, private 77%);

Perceived nurses role in advance directives

Majority agreed that nurses are crucial in educating about AD (public-78.1%, private-73.3%); in best position to access the appropriate time for end-of-life discussions (public-84.4%; private-78.6%) and are responsible to initiate end-of-life discussion (public-76%, private-57.3%). More participants in the private facilities than those at the public agreed that nurse can transfer a patient to another nurse when not comfortable with the directives.

Perceived shortcomings of advance directive

More participants in public (42.2%) than private (35.9%) disagreed that interpreting AD can be challenging. Two-thirds of private nurses (65.7%) and 55.5% of nurses in the public hospital agreed that AD can lead to requests for care not in the patient’s best interests. The little more than half of the participants felt AD might not accurately reflect patient’s current preferences (public-57.8%; private-52.7%) and uncertain (public-51.8%, private 39.7).

Classification of nurses responses into positive and negative perception

Table 3 shows how nurses generally perceived the benefits of AD, their involvement in its initiation and execution, and its perceived drawbacks. Majority of nurses in the public sector (75.2%) and private sector (72.5%) agreed AD is beneficial to patients, their families, and healthcare providers. More participants in the public sector (61.9%) than private (56.5%) thought they played a critical role in the development and implementation of AD. More nurses (60.7%) in the public sector concurred that AD had drawbacks than its private counterpart (58.8%).

Differences on nurses perception to advance directive

Table 4 below displays how Muslim nurses and Christian nurses perceive AD using odd ratios. Significantly, Christian nurses are 0.53 times less likely than Muslims to contest AD’s benefits; are 0.78 times less likely than Muslim to disagree that AD has flaws but are 1.30 times more likely than Muslim nurses to deny they contributed to the development and execution of AD, albeit, these differences are not statistically significant.

This study focused on nurses’ perceptions on the benefits, the role of nurses, and the negative aspects of AD at public and private secondary healthcare units in Ibadan, Nigeria. Positive perception regarding AD advantages was found among nurses at both public and private secondary healthcare units. This supported previous reported finding in Australia and Korea. According to these researches, AD guarantee patient autonomy, improve end-of-life care, and give patients a chance to reflect on their own dying stage and demise [ 22 , 23 , 24 ]. The study findings also agreed with prior researches where it was reported that the enforcement of ADs relieved families and patients’ financial, emotional weariness and disagreement [ 24 , 25 ] as we found that, participants agreed that AD can reduced needless stress, excessive spending and prevented or resolved conflict among healthcare practitioners, patients and patients relatives.

Nurses are more available at hospital and are closer to patients than any other healthcare practitioners. As a result, they agreed they are the best resource for patients and their families seeking information about AD. This support earlier researches in Portugal, Korea, New Zealand, and Australia [ 3 , 12 , 22 , 24 , 26 ]. The disparity reported on who is proficient in figuring out the appropriate time to initiate AD among the two group of nurses could be attributed to the quantity and quality of training enjoyed by these nurses. While more trainings are planned for nurses at the public sector, little of such training is available for nurses at the private sector in Nigeria. Davidson et al. also reported that nurses are in the best position to initiate AD [ 12 ]. On who should start the end-of-life conversation with a patient, the nurses at the two healthcare facilities had contrasting opinions. Nurses at private facilities saw it as the doctors’ obligation to begin and record the decision while they made the document readily available when needed, in contrast to nurses at public hospitals who saw it as their role. These was similar to findings in South Africa, Korea and Australia by Bull and Mash, Son et al., and Hobden et al. [ 24 , 27 , 28 ], where nurses saw themselves as the custodians of AD document rather than its initiators and/or implementers. The findings demonstrated that nurses in the private sector are more likely to refer patients whose orders they find objectionable to another nurse or facility. These both supports Siamak’s [ 23 ] findings that nurses have the autonomy to decline participation in the withdrawing or withholding of treatment if such a decision contradicts their personal and/or professional convictions and Hobden’s [ 27 ] findings where 60% of their study participants showed neutrality or disagreement that ADs will still be adhered to even if the medical team does not agree with them. Fear of litigation and the fact that nurses at the public sector enjoyed more autonomy, employment security, and public reputation than those in the private sector are some contributing factors to this [ 26 ]. Making known and reporting violation of patients’ directives were found to be nurses’ responsibilities in the present study. This was in consistent with Hobden et al. [ 27 ] that found nurses play a key role in ensuring that patients’ preferences are honored throughout end-of-life care.

Regarding AD’s shortcomings, the study demonstrates consensus that AD has some degree of negativity, but to various degrees. Over 50% of the study participants in the two sectors agreed and are neutral on the statement that many ambiguous terms are frequently used in AD without enough context or justification thereby making it difficult to interpret. Previous researchers have also noted that unclear instructions and the use of ambiguous language could lead to misreading of patients’ preferences [ 20 , 21 , 27 , 29 ]. The fact that patients’ mostly give their directives verbally when they are critically ill and sometime by their relatives in Nigeria can also contribute to the misapprehension of the directives [ 7 , 30 ]. More public sector nurses thought it was challenging to prove that AD is certain and accurately reflect patients’ current preferences and this made its implementation challenging. Reasons could be because, patients’ decisions regarding their treatment preference evolved over the illness episode due to factors like finance, relative decision, religious beliefs among others. However, these changes may not have reflected in the patient AD or known to the patient proxy. These contributed to the controversy in its implementation. Thus, decisional conflict that results from translating a written order into practice has previously been identified as an obstacle to the application of AD [ 26 , 27 , 28 ]. Ernestina et al. [ 3 ] showed that AD can fail in practice if changes in patient personal value fail to reflect in the directive. Therefore, AD should be periodically addressed and revisited for timely updates [ 21 ]. More than half of the study participants agreed that, there are chances that patients will asked for treatment that is not in their best interests in their AD. This finding supported researches conducted in Queensland, Australia, and Korea where it was reported that AD inhibited medical personnel from providing ethically and medically appropriate treatment to patient [ 20 , 26 , 29 ]. Inadequate knowledge and wrong cultural preconceptions about health, illness and treatment among patients could contributed to this perception. This study has been able to support existing knowledge that religion affiliation influence perception to end-of-life care [ 31 , 32 ]. While more Christian nurses are optimistic on the benefits of AD than Muslim nurses, more Muslim nurses believed they have a role to play in its initiation and execution than their Christians counterpart and thought AD had lesser flaws than the Christians. One of the tenets of Islam is to work for this life as if you were going to live forever and strive for the afterlife as if you were going to die tomorrow [ 33 ]. The Holy Qur’an also instructs Muslims to prepare and strategize their affairs. These may have influenced their perception that they have a greater role to play in the planning and implementation of the patient’s AD and support for AD. The Christian religion also supports AD as useful because it aids patients to avoid unbeneficial treatment [ 34 ].

In line with the theoretical explanation, the study had demonstrated that although AD has some benefits, such as quick decision-making, conflict resolution, and the prevention of wasteful spending; nurses as members of the healthcare team have a role to play in its initiation and implementation of AD. However, AD does have certain unintended consequences, which are its drawbacks [ 17 ].

This study has added to the corpus of research by identifying the perception of AD at the secondary healthcare facility in Nigeria and the chance that it will be adopted by nurses, who make up the majority of healthcare professionals. The study is limited by the use of the Likert scale to score nurses’ perceptions, which might have inhibited participants from fully expressing their perspectives on the matter. Further research should look into the acceptance of AD among terminally ill patients and their families as well as the amount of abuse or improper inducement of AD among healthcare professionals in secondary and tertiary healthcare facilities in Nigeria.

Making decisions in the final stages of life might be challenging, however AD may make these challenges easier. As a result, AD should be acknowledged in all healthcare sectors as a tool capable of granting patients’ liberty and dignity. Both in the public secondary healthcare unit and the private unit, nurses play a vital role as care providers in the development and execution of patient Ads. However, some of the difficulties in implementing AD that have been identified in this study should be addressed by stakeholders, and nurses at both sectors should be provided with necessary training on how to avoid these difficulties.

Availability of data and materials

Due to confidentiality rules, the datasets created and/or analyzed for the current work are not publically accessible, but they are available from the corresponding author upon justifiable request.

Teixeira C, Ribeiro O, Fonseca A, Carvalho AS. Ethical decision making in intensive care units: A burnout risk factor? Results from a multicenter study conducted with physicians and nurses. J Med Ethics. 2014;40:97–103.

Article Google Scholar

Dijxhoorn AF, Brom L, Van der Linden Y, Leget C, Raijmakers N. Prevalence of burnout in healthcare professionals providing palliative care and the effect of interventions to reduce symptoms: a systematic literature review. Palliat Med. 2021;35(1):6–26.

Ernestina S, Maria N, Daniel S. Nurses’ perceptions of advance directives. researchgate.net/publication/ 2014;269:187–357.

Izumi S. Advance care planning: the nurse’s role. Am J Nurs. 2017;117(6):56–61.

Ekore RI, Lanre-Abass B. African cultural concept of death and the idea of advance care directives. Indian J Palliat Care. 2016;22(4):369–72.

American Cancer Society (ACS). Types of advance directives. 2015. Retrieved from http://www.cancer.org/treatment/findingandpayingfortreatment/understandingfinancialandlegalmatters/advancedirectives/advance-directives-types-of-advancehealth-care-directives

Jegede AS, Adegoke OO. Advance directive in end-of-life decision-making among the Yoruba of South-Western Nigeria. BEOnline J West Afr Bioethics Train Prog. 2016;3(3):41–67.

Google Scholar

Hong JH, Kwon JH, Kim IK, Ko JH, Kang YJ, Kim HK. Adopting advance directives reinforces patient participation in end-of-life care discussion. Cancer Res Treat. 2016;48(2):753–8. https://doi.org/10.4143/crt.2015.281 .

Innocent EO, Uche OA, Uche IB. Building a solid health care system in Nigeria: challenges and prospects. Acad J Interdiscip Stud. 2014;3(6):501–10. https://doi.org/10.5901/ajis.2014.v3n6p501 .

Koce F, Randhawa G, Ochieng B. Understanding healthcare self-referral in Nigeria from the service users’ perspective: a qualitative study of Niger state. BMC Health Serv Res. 2019;19:209.

Ofoli JNT, Ashau-Oladipo T, Hati SS, et al. Preventive healthcare uptake in private hospitals in Nigeria: a cross-sectional survey (Nisa premier hospital). BMC Health Serv Res. 2020;20:273.

Davidson R, Banister E, de Vries K. Primary healthcare NZ nurses’ experiences of advance directives: understanding their potential role. Nurs Prax N Z. 2013;29(2):26–33.

Oldland E, Botti M, Hutchinson AM, Redley B. A framework of nurses’ responsibilities for quality healthcare—exploration of content validity. Collegian. 2020;27(2):150–63.

Ke LS, Huang X, O’Connor M, Lee S. Nurses’ views regarding implementing advance care planning for older people: a systematic review and synthesis of qualitative studies. J Clin Nurs. 2015;24(15–16):2057–73.

Nguyen KT. Exploring patient perspective at end of life: qualitative interviews with terminally Ill patients. Yale; 2012.

Amboro JL, Wahyuningsih N, Prameswari NS. A structural functionalism point of view: ritual ceremony philosophy of Grebeg Maulud at Kasunanan Palace, Surakarta. J Adv Humanit Soc Sci. 2020;6(2):71–7.

Longhofer W, Winchester D. Social theory re-wired: new connections to classical and contemporary perspective. 2nd ed. New York: Routledge; 2016.

Book Google Scholar

C. A. Wojuade and S. O. Fadare. Research on Humanities and Social Sciences. 2014;4(6). www.iiste.org ISSN (Paper)2224-5766 ISSN (Online)2225-0484 (Online).

Makinde OA, Sule A, Ayankogbe O, Boone D. Distribution of health facilities in Nigeria: implications and options for Universal Health Coverage. Int J Health Plan Manag. 2018;33(4):e1179–92. https://doi.org/10.1002/hpm.2603 .

Lindy W, Ben W, Cheryl T, Jill W, Kelly P. Advance health directives: competing perceptions, intentions and use by patients and doctors in Queensland. QUT Law Rev. 2013;13(1):520–32.

Peicius E, Blazeviciene A, Kaminskas R. are advance directive helpful for good end of life decision making: a cross sectional survey of health professionals. BMC Med Ethnics. 2017;18(1):40.

Johnson C, Singer R, Masso M, Sellars M, Silvester W. Palliative care health professionals’ experiences of caring for patients with advance care directives. Aust Health. 2015;39(2):154–9.

Siamak NN. Advance medical directives (living will, power of attorney and health-care proxy). Medicine Net. 2016

Son Y, Choib J, Ahn J. Nurses’ perspectives on advance directives before the establishment of the new well-dying law in Korea: a mixed methods study. Appl Nurs Res. 2019;51:151187.

Donnelly LK. Increasing advance directive knowledge among healthcare professionals. Family Medicine Block Clerkship, Student Projects. 2017;238.

Poveda-Mora S, Rodriguez-Martin D, Codern-Bove N, Jose-Maria P, Sanchez-Valero P, Pomares-Quintana N, et al. Managing ethical aspects of advance directives in emergency care services. Nurs Ethics. 2021;28(1):91–105. https://doi.org/10.1177/0969733020952112 .

Hobden B, Bryant J, Waller A, Shepherd J, Sanson-Fisher R. Oncology nurses’ perceptions of advance directives for patients with cancer. Nurs Health Sci. 2021;23:1–8.

Bull A, Mash B. Advance Directives or Living Wills: Reflections of general practitioners and frail care coordinators in a small town in KwaZulu-Natal. S Afr Acad Fam Pract Prim Care. 2014;54(6):507–12.

Moore N, Detering KM, Low T, Nolte L, Fraser S, Sellars M. Doctors’ perspectives on adhering to advance care directives when making medical decisions for patients: an Australian interview study. BMJ Open. 2019;9:e032638.

Fadare JO, Obimakinde AM, Olaogun DO, Afolayan JM, Olatunya O, Ogundipe KO. Perception of nurses about palliative care: Experience from South-West Nigeria. Ann Med Health Sci Res. 2014;4(5):723. https://doi.org/10.4103/2141-9248.141532 .

Abudari G, Hazeim H, Ginete A. Caring for terminally ill Muslim patients: LIVED experiences of non-Muslim nurses. Palliat Support Care. 2016;14:599–611. https://doi.org/10.1017/S1478951516000249 .

Seale C. The role of doctors’ religious faith and ethnicity in taking ethically controversial decisions during end-of-life care. J Med Ethics. 2010;36:677–82. https://doi.org/10.1136/jme.2010.036194 .

Khan FA. Khan. 2022. Religious teachings and reflections on advance directive—religious values and legal dilemmas in bioethics: an Islamic perspective, 30 Fordham Urb. L.J. 267. 2002. Available at: https://ir.lawnet.fordham.edu/ulj/vol30/iss1/1 .

Kaplan RL. Religion and advance medical directives: formulation and enforcement implications. University Of Illinois Law Review; 2016. Accessed from: https://polstfl.org/wp-content/uploads/2017/12/Religion-and-Advance-Directives.pd .

Download references

Acknowledgements

The researchers appreciate the effort and interest of all the nurses who took part in the study, as well as the thoughtful criticism provided by the anonymous reviewers.

The article was not funded in any way.

Author information

Authors and affiliations.

University of Ibadan, Ibadan, Nigeria

Oluwaseyi Emiola Ojedoyin & Ayodele Samuel Jegede

You can also search for this author in PubMed Google Scholar

Contributions

OO made contributions to the concept, literature review, design, data collection, analysis, and findings discussion. AJ oversaw the study, contributed to the concept and design, and critically examined the report for key intellectual content. The final manuscript was read and approved by all writers.

Corresponding author

Correspondence to Oluwaseyi Emiola Ojedoyin .

Ethics declarations

Ethics approval and consent to participate.

The Oyo State Research Ethics Review Committee, with reference number AD13/479/837, as well as the administrative head in each of the chosen hospitals and each participant, all approved this study. Every approach used in the study complied with the rules and regulations established by the institutional Research Committee for research involving people. All individuals participated in the study provided their written, informed consent.

Consent for publication

Not applicable.

Competing interests

No potential conflicts of interest were disclosed by the authors with regard to the research, writing, or publication of this paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ojedoyin, O.E., Jegede, A.S. A quantitative study of nurses perception to advance directive in selected private and public secondary healthcare facilities in Ibadan, Nigeria. BMC Med Ethics 23 , 87 (2022). https://doi.org/10.1186/s12910-022-00825-5

Download citation

Received : 07 February 2022

Accepted : 16 August 2022

Published : 25 August 2022

DOI : https://doi.org/10.1186/s12910-022-00825-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Treatment preference
End-of-life care
Living will
General hospital
Private hospital

BMC Medical Ethics

ISSN: 1472-6939

General enquiries: [email protected]

Systematic review
Open access
Published: 19 June 2020

Quantitative measures of health policy implementation determinants and outcomes: a systematic review

Peg Allen ORCID: orcid.org/0000-0001-7000-796X 1 ,
Meagan Pilar 1 ,
Callie Walsh-Bailey 1 ,
Cole Hooley 2 ,
Stephanie Mazzucca 1 ,
Cara C. Lewis 3 ,
Kayne D. Mettert 3 ,
Caitlin N. Dorsey 3 ,
Jonathan Purtle 4 ,
Maura M. Kepper 1 ,
Ana A. Baumann 5 &
Ross C. Brownson 1 , 6

Implementation Science volume 15 , Article number: 47 ( 2020 ) Cite this article

19k Accesses

62 Citations

31 Altmetric

Metrics details

Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures.

Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures.

Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported.

Conclusions

Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health.

Registration

Not registered

Peer Review reports

Contributions to the literature

This systematic review identified 70 quantitative measures of implementation outcomes or determinants in health policy studies.

Readiness to implement and organizational climate and culture were commonly assessed determinants, but fewer studies assessed policy actor relationships or implementation outcomes of acceptability, fidelity/compliance, appropriateness, feasibility, or implementation costs.

Study team members rated most identified measures’ pragmatic properties as good, meaning they are straightforward to use, but few studies documented pilot or psychometric testing of measures.

Further development and dissemination of valid and reliable measures of policy implementation outcomes and determinants can facilitate identification, use, and spread of effective policy implementation strategies.

Despite major impacts of policy on population health [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ], there have been relatively few policy studies in dissemination and implementation (D&I) science to inform implementation strategies and evaluate implementation efforts [ 8 ]. While health outcomes of policies are commonly studied, fewer policy studies assess implementation processes and outcomes. Of 146 D&I studies funded by the National Institutes of Health (NIH) through D&I funding announcements from 2007 to 2014, 12 (8.2%) were policy studies that assessed policy content, policy development processes, or health outcomes of policies, representing 10.5% of NIH D&I funding [ 8 ]. Eight of the 12 studies (66.7%) assessed health outcomes, while only five (41.6%) assessed implementation [ 8 ].

Our ability to explore the differential impact of policy implementation determinants and outcomes and disentangle these from health benefits and other societal outcomes requires high quality quantitative measures [ 9 ]. While systematic reviews of measures of implementation of evidence-based interventions (in clinical and community settings) have been conducted in recent years [ 10 , 11 , 12 , 13 ], to our knowledge, no reviews have explored the quality of quantitative measures of determinants and outcomes of policy implementation.

Policy implementation research in political science and the social sciences has been active since at least the 1970s and has much to contribute to the newer field of D&I research [ 1 , 14 ]. Historically, theoretical frameworks and policy research largely emphasized policy development or analysis of the content of policy documents themselves [ 15 ]. For example, Kingdon’s Multiple Streams Framework and its expansions have been widely used in political science and the social sciences more broadly to describe how factors related to sociopolitical climate, attributes of a proposed policy, and policy actors (e.g., organizations, sectors, individuals) contribute to policy change [ 16 , 17 , 18 ]. Policy frameworks can also inform implementation planning and evaluation in D&I research. Although authors have named policy stages since the 1950s [ 19 , 20 ], Sabatier and Mazmanian’s Policy Implementation Process Framework was one of the first such frameworks that gained widespread use in policy implementation research [ 21 ] and later in health promotion [ 22 ]. Yet, available implementation frameworks are not often used to guide implementation strategies or inform why a policy worked in one setting but not another [ 23 ]. Without explicit focus on implementation, the intended benefits of health policies may go unrealized, and the ability may be lost to move the field forward to understand policy implementation (i.e., our collective knowledge building is dampened) [ 24 ].

Differences in perspectives and terminology between D&I and policy research in political science are noteworthy to interpret the present review. For example, Proctor et al. use the term implementation outcomes for what policy researchers call policy outputs [ 14 , 20 , 25 ]. To non-D&I policy researchers, policy implementation outcomes refer to the health outcomes in the target population [ 20 ]. D&I science uses the term fidelity [ 26 ]; policy researchers write about compliance [ 20 ]. While D&I science uses the terms outer setting, outer context, or external context to point to influences outside the implementing organization [ 26 , 27 , 28 ], non-D&I policy research refers to policy fields [ 24 ] which are networks of agencies that carry out policies and programs.

Identification of valid and reliable quantitative measures of health policy implementation processes is needed. These measures are needed to advance from classifying constructs to understanding causality in policy implementation research [ 29 ]. Given limited resources, policy implementers also need to know which aspects of implementation are key to improve policy acceptance, compliance, and sustainability to reap the intended health benefits [ 30 ]. Both pragmatic and psychometrically sound measures are needed to accomplish these objectives [ 10 , 11 , 31 , 32 ], so the field can explore the influence of nuanced determinants and generate reliable and valid findings.

To fill this void in the literature, this systematic review of health policy implementation measures aimed to (1) identify quantitative measures used to assess health policy implementation outcomes (IOF outcomes commonly called policy outputs in policy research) and inner and outer setting determinants, (2) describe and assess pragmatic quality of policy implementation measures, (3) describe and assess the quality of psychometric properties of identified instruments, and (4) elucidate health policy implementation measurement gaps.

The study team used systematic review procedures developed by Lewis and colleagues for reviews of D&I research measures and received detailed guidance from the Lewis team coauthors for each step [ 10 , 11 ]. We followed the PRISMA reporting guidelines as shown in the checklist (Supplemental Table 1 ). We have also provided a publicly available website of measures identified in this review ( https://www.health-policy-measures.org/ ).

For the purposes of this review, policy and policy implementation are defined as follows. We deemed public policy to include legislation at the federal, state/province/regional unit, or local levels; and governmental regulations, whether mandated by national, state/province, or local level governmental agencies or boards of elected officials (e.g., state boards of education in the USA) [ 4 , 20 ]. Here, public policy implementation is defined as the carrying out of a governmental mandate by public or private organizations and groups of organizations [ 20 ].

Two widely used frameworks from the D&I field guide the present review, and a third recently developed framework that bridges policy and D&I research. In the Implementation Outcomes Framework (IOF), Proctor and colleagues identify and define eight implementation outcomes that are differentiated from health outcomes: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability [ 25 ]. In the Consolidated Framework for Implementation Research (CFIR), Damschroder and colleagues articulate determinants of implementation including the domains of intervention characteristics, outer setting, inner setting of an organization, characteristics of individuals within organizations, and process [ 33 ]. Finally, Bullock developed the Policy Implementation Determinants Framework to present a balanced framework that emphasizes both internal setting constructs and external setting constructs including policy actor relationships and networks, political will for implementation, and visibility of policy actors [ 34 ]. The constructs identified in these frameworks were used to guide our list of implementation determinants and outcomes.

Through EBSCO, we searched MEDLINE, PsycInfo, and CINAHL Plus. Through ProQuest, we searched PAIS, Worldwide Political, and ERIC. Due to limited time and staff in the 12-month study, we did not search the grey literature. We used multiple search terms in each of four required levels: health, public policy, implementation, and measurement (Table 1 ). Table 1 shows search terms for each string. Supplemental Tables 2 and 3 show the final search syntax applied in EBSCO and ProQuest.

The authors developed the search strings and terms based on policy implementation framework reviews [ 34 , 35 ], additional policy implementation frameworks [ 21 , 22 ], labels and definitions of the eight implementation outcomes identified by Proctor et al. [ 25 ], CFIR construct labels and definitions [ 9 , 33 ], and additional D&I research and search term sources [ 28 , 36 , 37 , 38 ] (Table 1 ). The full study team provided three rounds of feedback on draft terms, and a library scientist provided additional synonyms and search terms. For each test search, we calculated the percentage of 18 benchmark articles the search captured. We determined a priori 80% as an acceptable level of precision.

Inclusion and exclusion criteria

This review addressed only measures of implementation by organizations mandated to act by governmental units or legislation. Measures of behavior changes by individuals in target populations as a result of legislation or governmental regulations and health status changes were outside the realm of this review.

There were several inclusion criteria: (1) empirical studies of the implementation of public policies already passed or approved that addressed physical or behavioral health, (2) quantitative self-report or archival measurement methods utilized, (3) published in peer-reviewed journals from January 1995 through April 2019, (4) published in the English language, (5) public policy implementation studies from any continent or international governing body, and (6) at least two transferable quantitative self-report or archival items that assessed implementation determinants [ 33 , 34 ] and/or IOF implementation outcomes [ 25 ]. This study sought to identify transferable measures that could be used to assess multiple policies and contexts. Here, a transferable item is defined as one that needed no wording changes or only a change in the referent (e.g., policy title or topic such as tobacco or malaria) to make the item applicable to other policies or settings [ 11 ]. The year 1995 was chosen as a starting year because that is about when web-based quantitative surveying began [ 39 ]. Table 2 provides definitions of the IOF implementation outcomes and the selected determinants of implementation. Broader constructs, such as readiness for implementation, contained multiple categories.

Exclusion criteria in the searches included (1) non-empiric health policy journal articles (e.g., conceptual articles, editorials); (2) narrative and systematic reviews; (3) studies with only qualitative assessment of health policy implementation; (4) empiric studies reported in theses and books; (5) health policy studies that only assessed health outcomes (i.e., target population changes in health behavior or status); (6) bill analyses, stakeholder perceptions assessed to inform policy development, and policy content analyses without implementation assessment; (7) studies of changes made in a private business not encouraged by public policy; and (8) countries with authoritarian regimes. We electronically programmed the searches to exclude policy implementation studies from countries that are not democratically governed due to vast differences in policy environments and implementation factors.

Screening procedures

Citations were downloaded into EndNote version 7.8 and de-duplicated electronically. We conducted dual independent screening of titles and abstracts after two group pilot screening sessions in which we clarified inclusion and exclusion criteria and screening procedures. Abstract screeners used Covidence systematic review software [ 40 ] to code inclusion as yes or no. Articles were included in full-text review if one screener coded it as meeting the inclusion criteria. Full-text screening via dual independent screening was coded in Covidence [ 40 ], with weekly meetings to reach consensus on inclusion/exclusion discrepancies. Screeners also coded one of the pre-identified reasons for exclusion.

Data extraction strategy

Extraction elements included information about (1) measure meta-data (e.g., measure name, total number of items, number of transferable items) and studies (e.g., policy topic, country, setting), (2) development and testing of the measure, (3) implementation outcomes and determinants assessed (Table 2 ), (4) pragmatic characteristics, and (5) psychometric properties. Where needed, authors were emailed to obtain the full measure and measure development information. Two coauthors (MP, CWB) reached consensus on extraction elements. For each included measure, a primary extractor conducted initial entries and coding. Due to time and staff limitations in the 12-month study, we did not search for each empirical use of the measure. A secondary extractor checked the entries, noting any discrepancies for discussion in consensus meetings. Multiple measures in a study were extracted separately.

Quality assessment of measures

To assess the quality of measures, we applied the Psychometric and Pragmatic Evidence Rating Scales (PAPERS) developed by Lewis et al. [ 10 , 11 , 41 , 42 ]. PAPERS includes assessment of five pragmatic instrument characteristics that affect the level of ease or difficulty to use the instrument: brevity (number of items), simplicity of language (readability level), cost (whether it is freely available), training burden (extent of data collection training needed), and analysis burden (ease or difficulty of interpretation of scoring and results). Lewis and colleagues developed the pragmatic domains and rating scales with stakeholder and D&I researchers input [ 11 , 41 , 42 ] and developed the psychometric rating scales in collaboration with D&I researchers [ 10 , 11 , 43 ]. The psychometric rating scale has nine properties (Table 3 ): internal consistency; norms; responsiveness; convergent, discriminant, and known-groups construct validity; predictive and concurrent criterion validity; and structural validity. In both the pragmatic and psychometric scales, reported evidence for each domain is scored from poor (− 1), none/not reported (0), minimal/emerging (1), adequate (2), good (3), or excellent (4). Higher values are indicative of more desirable pragmatic characteristics (e.g., fewer items, freely available, scoring instructions, and interpretations provided) and stronger evidence of psychometric properties (e.g., adequate to excellent reliability and validity) (Supplemental Tables 4 and 5 ).

Data synthesis and presentation

This section describes the synthesis of measure transferability, empiric use study settings and policy topics, and PAPERS scoring. Two coauthors (MP, CWB) consensus coded measures into three categories of item transferability based on quartile item transferability percentages: mostly transferable (≥ 75% of items deemed transferable), partially transferable (25–74% of items deemed transferable), and setting-specific (< 25% of items deemed transferable). Items were deemed transferable if no wording changes or only a change in the referent (e.g., policy title or topic) was needed to make the item applicable to the implementation of other policies or in other settings. Abstractors coded study settings into one of five categories: hospital or outpatient clinics; mental or behavioral health facilities; healthcare cost, access, or quality; schools; community; and multiple. Abstractors also coded policy topics to healthcare cost, access, or quality; mental or behavioral health; infectious or chronic diseases; and other, while retaining documentation of subtopics such as tobacco, physical activity, and nutrition. Pragmatic scores were totaled for the five properties, with possible total scores of − 5 to 20, with higher values indicating greater ease to use the instrument. Psychometric property total scores for the nine properties were also calculated, with possible scores of − 9 to 36, with higher values indicating evidence of multiple types of validity.

The database searches yielded 11,684 articles, of which 3267 were duplicates (Fig. 1 ). Titles and abstracts of the 8417 articles were independently screened by two team members; 870 (10.3%) were selected for full-text screening by at least one screener. Of the 870 studies, 804 were excluded at full-text screening or during extraction attempts with the consensus of two coauthors; 66 studies were included. Two coauthors (MP, CWB) reached consensus on extraction and coding of information on 70 unique quantitative eligible measures identified in the 66 included studies plus measure development articles where obtained. Nine measures were used in more than one included study. Detailed information on identified measures is publicly available at https://www.health-policy-measures.org/ .

PRISMA flow diagram

The most common exclusion reason was lack of transferable items in quantitative measures of policy implementation ( n = 597) (Fig. 1 ). While this review focused on transferable measures across any health issue or setting, researchers addressing specific health policies or settings may find the excluded studies of interest. The frequencies of the remaining exclusion reasons are listed in Fig. 1 .

A variety of health policy topics and settings from over two dozen countries were found in the database searches. For example, the searches identified quantitative and mixed methods implementation studies of legislation (such as tobacco smoking bans), regulations (such as food/menu labeling requirements), governmental policies that mandated specific clinical practices (such as vaccination or access to HIV antiretroviral treatment), school-based interventions (such as government-mandated nutritional content and physical activity), and other public policies.

Among the 70 unique quantitative implementation measures, 15 measures were deemed mostly transferable (at least 75% transferable, Table 4 ). Twenty-three measures were categorized as partially transferable (25 to 74% of items deemed transferable, Table 5 ); 32 measures were setting-specific (< 25% of items deemed transferable, data not shown).

Implementation outcomes

Among the 70 measures, the most commonly assessed implementation outcomes were fidelity/compliance of the policy implementation to the government mandate (26%), acceptability of the policy to implementers (24%), perceived appropriateness of the policy (17%), and feasibility of implementation (17%) (Table 2 ). Fidelity/compliance was sometimes assessed by asking implementers the extent to which they had modified a mandated practice [ 45 ]. Sometimes, detailed checklists were used to assess the extent of compliance with the many mandated policy components, such as school nutrition policies [ 83 ]. Acceptability was assessed by asking staff or healthcare providers in implementing agencies their level of agreement with the provided statements about the policy mandate, scored in Likert scales. Only eight (11%) of the included measures used multiple transferable items to assess adoption, and only eight (11%) assessed penetration.

Twenty-six measures of implementation costs were found during full-text screening (10 in included studies and 14 in excluded studies, data not shown). The cost time horizon varied from 12 months to 21 years, with most cost measures assessed at multiple time points. Ten of the 26 measures addressed direct implementation costs. Nine studies reported cost modeling findings. The implementation cost survey developed by Vogler et al. was extensive [ 53 ]. It asked implementing organizations to note policy impacts in medication pricing, margins, reimbursement rates, and insurance co-pays.

Determinants of implementation

Within the 70 included measures, the most commonly assessed implementation determinants were readiness for implementation (61% assessed any readiness component) and the general organizational culture and climate (39%), followed by the specific policy implementation climate within the implementation organization/s (23%), actor relationships and networks (17%), political will for policy implementation (11%), and visibility of the policy role and policy actors (10%) (Table 2 ). Each component of readiness for implementation was commonly assessed: communication of the policy (31%, 22 of 70 measures), policy awareness and knowledge (26%), resources for policy implementation (non-training resources 27%, training 20%), and leadership commitment to implement the policy (19%).

Only two studies assessed organizational structure as a determinant of health policy implementation. Lavinghouze and colleagues assessed the stability of the organization, defined as whether re-organization happens often or not, within a set of 9-point Likert items on multiple implementation determinants designed for use with state-level public health practitioners, and assessed whether public health departments were stand-alone agencies or embedded within agencies addressing additional services, such as social services [ 69 ]. Schneider and colleagues assessed coalition structure as an implementation determinant, including items on the number of organizations and individuals on the coalition roster, number that regularly attend coalition meetings, and so forth [ 72 ].

Tables of measures

Tables 4 and 5 present the 38 measures of implementation outcomes and/or determinants identified out of the 70 included measures with at least 25% of items transferable (useable in other studies without wording changes or by changing only the policy name or other referent). Table 4 shows 15 mostly transferable measures (at least 75% transferable). Table 5 shows 23 partially transferable measures (25–74% of items deemed transferable). Separate measure development articles were found for 20 of the 38 measures; the remaining measures seemed to be developed for one-time, study-specific use by the empirical study authors cited in the tables. Studies listed in Tables 4 and 5 were conducted most commonly in the USA ( n = 19) or Europe ( n = 11). A few measures were used elsewhere: Africa ( n = 3), Australia ( n = 1), Canada ( n = 1), Middle East ( n = 1), Southeast Asia ( n = 1), or across multiple continents ( n = 1).

Quality of identified measures

Figure 2 shows the median pragmatic quality ratings across the 38 measures with at least 25% transferable items shown in Tables 4 and 5 . Higher scores are desirable and indicate the measures are easier to use (Table 3 ). Overall, the measures were freely available in the public domain (median score = 4), brief with a median of 11–50 items (median score = 3), and had good readability, with a median reading level between 8th and 12th grade (median score = 3). However, instructions on how to score and interpret item scores were lacking, with a median score of 1, indicating the measures did not include suggestions for interpreting score ranges, clear cutoff scores, and instructions for handling missing data. In general, information on training requirements or availability of self-training manuals on how to use the measures was not reported in the included study or measure development article/s (median score = 0, not reported). Total pragmatic rating scores among the 38 measures with at least 25% of items transferable ranged from 7 to 17 (Tables 4 and 5 ), with a median total score of 12 out of a possible total score of 20. Median scores for each pragmatic characteristic were the same across all measures as for the 38 mostly or partially transferable measures, with a median total score of 11 across all measures.

Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [ 11 ], Stanick et al. [ 42 ]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4

Few psychometric properties were reported. The study team found few reports of pilot testing and measure refinement as well. Among the 38 measures with at least 25% transferable items, the psychometric properties from the PAPERS rating scale total scores ranged from − 1 to 17 (Tables 4 and 5 ), with a median total score of 5 out of a possible total score of 36. Higher scores indicate more types of validity and reliability were reported with high quality. The 32 measures with calculable norms had a median norms PAPERS score of 3 (good), indicating appropriate sample size and distribution. The nine measures with reported internal consistency mostly showed Cronbach’s alphas in the adequate (0.70 to 0.79) to excellent (≥ 90) range, with a median of 0.78 (PAPERS score of 2, adequate) indicating adequate internal consistency. The five measures with reported structural validity had a median PAPERS score of 2, adequate (range 1 to 3, poor to good), indicating the sample size was sufficient and the factor analysis goodness of fit was reasonable. Among the 38 measures, no reports were found for responsiveness, convergent validity, discriminant validity, known-groups construct validity, or predictive or concurrent criterion validity.

In this systematic review, we sought to identify quantitative measures used to assess health policy implementation outcomes and determinants, rate the pragmatic and psychometric quality of identified measures, and point to future directions to address measurement gaps. In general, the identified measures are easy to use and freely available, but we found little data on validity and reliability. We found more quantitative measures of intra-organizational determinants of policy implementation than measures of the relationships and interactions between organizations that influence policy implementation. We found a limited number of measures that had been developed for or used to assess one of the eight IOF policy implementation outcomes that can be applied to other policies or settings, which may speak more to differences in terms used by policy researchers and D&I researchers than to differences in conceptualizations of policy implementation. Authors used a variety of terms and rarely provided definitions of the constructs the items assessed. Input from experts in policy implementation is needed to better understand and define policy implementation constructs for use across multiple fields involved in policy-related research.

We found several researchers had used well-tested measures of implementation determinants from D&I research or from organizational behavior and management literature (Tables 4 and 5 ). For internal setting of implementing organizations, whether mandated through public policy or not, well-developed and tested measures are available. However, a number of authors crafted their own items, with or without pilot testing, and used a variety of terms to describe what the items assessed. Further dissemination of the availability of well-tested measures to policy researchers is warranted [ 9 , 13 ].

What appears to be a larger gap involves the availability of well-developed and tested quantitative measures of the external context affecting policy implementation that can be used across multiple policy settings and topics [ 9 ]. Lack of attention to how a policy initiative fits with the external implementation context during policymaking and lack of policymaker commitment of adequate resources for implementation contribute to this gap [ 23 , 93 ]. Recent calls and initiatives to integrate health policies during policymaking and implementation planning will bring more attention to external contexts affecting not only policy development but implementation as well [ 93 , 94 , 95 , 96 , 97 , 98 , 99 ]. At the present time, it is not well-known which internal and external determinants are most essential to guide and achieve sustainable policy implementation [ 100 ]. Identification and dissemination of measures that assess factors that facilitate the spread of evidence-informed policy implementation (e.g., relative advantage, flexibility) will also help move policy implementation research forward [ 1 , 9 ].

Given the high potential population health impact of evidence-informed policies, much more attention to policy implementation is needed in D&I research. Few studies from non-D&I researchers reported policy implementation measure development procedures, pilot testing, scoring procedures and interpretation, training of data collectors, or data analysis procedures. Policy implementation research could benefit from the rigor of D&I quantitative research methods. And D&I researchers have much to learn about the contexts and practical aspects of policy implementation and can look to the rich depth of information in qualitative and mixed methods studies from other fields to inform quantitative measure development and testing [ 101 , 102 , 103 ].

Limitations

This systematic review has several limitations. First, the four levels of the search string and multiple search terms in each level were applied only to the title, abstract, and subject headings, due to limitations of the search engines, so we likely missed pertinent studies. Second, a systematic approach with stakeholder input is needed to expand the definitions of IOF implementation outcomes for policy implementation. Third, although the authors value intra-organizational policymaking and implementation, the study team restricted the search to governmental policies due to limited time and staffing in the 12-month study. Fourth, by excluding tools with only policy-specific implementation measures, we excluded some well-developed and tested instruments in abstract and full-text screening. Since only 12 measures had 100% transferable items, researchers may need to pilot test wording modifications of other items. And finally, due to limited time and staffing, we only searched online for measures and measures development articles and may have missed separately developed pragmatic information, such as training and scoring materials not reported in a manuscript.

Despite the limitations, several recommendations for measure development follow from the findings and related literature [ 1 , 11 , 20 , 35 , 41 , 104 ], including the need to (1) conduct systematic, mixed-methods procedures (concept mapping, expert panels) to refine policy implementation outcomes, (2) expand and more fully specify external context domains for policy implementation research and evaluation, (3) identify and disseminate well-developed measures for specific policy topics and settings, (4) ensure that policy implementation improves equity rather than exacerbating disparities [ 105 ], and (5) develop evidence-informed policy implementation guidelines.

Easy-to-use, reliable, and valid quantitative measures of policy implementation can further our understanding of policy implementation processes, determinants, and outcomes. Due to the wide array of health policy topics and implementation settings, sound quantitative measures that can be applied across topics and settings will help speed learnings from individual studies and aid in the transfer from research to practice. Quantitative measures can inform the implementation of evidence-informed policies to further the spread and effective implementation of policies to ultimately reap greater population health benefit. This systematic review of measures is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and predictors to help practitioners and researchers spread evidence-informed policies to improve population health and reduce inequities.

Availability of data and materials

A compendium of identified measures is available for dissemination at https://www.health-policy-measures.org/ . A link will be provided on the website of the Prevention Research Center, Brown School, Washington University in St. Louis, at https://prcstl.wustl.edu/ . The authors invite interested organizations to provide a link to the compendium. Citations and abstracts of excluded policy-specific measures are available on request.

Abbreviations

Consolidated Framework for Implementation Research

Cumulative Index of Nursing and Allied Health Literature

Dissemination and implementation science

Elton B. Stephens Company

Education Resources Information Center

Implementation Outcomes Framework

Psychometric and Pragmatic Evidence Rating Scale

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Purtle J, Dodson EA, Brownson RC. Policy dissemination research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Google Scholar

Brownson RC, Baker EA, Deshpande AD, Gillespie KN. Evidence-based public health. Third ed. New York, NY: Oxford University Press; 2018.

Guide to Community Preventive Services. About the community guide.: community preventive services task force; 2020 [updated October 03, 2019; cited 2020. Available from: https://www.thecommunityguide.org/ .

Eyler AA, Chriqui JF, Moreland-Russell S, Brownson RC, editors. Prevention, policy, and public health, first edition. New York, NY: Oxford University Press; 2016.

Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ, et al. Vaccination greatly reduces disease, disability, death, and inequity worldwide. Geneva, Switzerland: World Health Organization; 2008 February 2008. Contract No.: 07-040089.

Cheng JJ, Schuster-Wallace CJ, Watt S, Newbold BK, Mente A. An ecological quantification of the relationships between water, sanitation and infant, child, and maternal mortality. Environ Health. 2012;11:4.

PubMed PubMed Central Google Scholar

Levy DT, Li Y, Yuan Z. Impact of nations meeting the MPOWER targets between 2014 and 2016: an update. Tob Control. 2019.

Purtle J, Peters R, Brownson RC. A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007-2014. Implement Sci. 2016;11:1.

Lewis CC, Proctor EK, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Ga C, Proctor EK, editors. Disssemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155.

Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, et al. An updated protocol for a systematic review of implementation-related measures. Syst Rev. 2018;7(1):66.

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.

Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, et al. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11:42.

Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet?--a comparison of implementation science and policy implementation research. Implement Sci. 2013;8:63.

Sabatier PA, editor. Theories of the Policy Process. New York, NY: Routledge; 2019.

Kingdon J. Agendas, alternatives, and public policies, second edition. Second ed. New York: Longman; 1995.

Jones MD, Peterson HL, Pierce JJ, Herweg N, Bernal A, Lamberta Raney H, et al. A river runs through it: a multiple streams meta-review. Policy Stud J. 2016;44(1):13–36.

Fowler L. Using the multiple streams framework to connect policy adoption to implementation. Policy Studies Journal. 2020 (11 Feb).

Howlett M, Mukherjee I, Woo JJ. From tools to toolkits in policy design studies: the new design orientation towards policy formulation research. Policy Polit. 2015;43(2):291–311.

Natesan SD, Marathe RR. Literature review of public policy implementation. Int J Public Policy. 2015;11(4):219–38.

Sabatier PA, Mazmanian. Implementation of public policy: a framework of analysis. Policy Studies Journal. 1980 (January).

Sabatier PA. Theories of the Policy Process. Westview; 2007.

Tomm-Bonde L, Schreiber RS, Allan DE, MacDonald M, Pauly B, Hancock T, et al. Fading vision: knowledge translation in the implementation of a public health policy intervention. Implement Sci. 2013;8:59.

Roll S, Moulton S, Sandfort J. A comparative analysis of two streams of implementation research. Journal of Public and Nonprofit Affairs. 2017;3(1):3–22.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.

Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice, second edition. New York: Oxford University Press; 2018.

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.

Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23.

PubMed Google Scholar

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Boyd MR, Powell BJ, Endicott D, Lewis CC. A method for tracking implementation strategies: an exemplar implementing measurement-based care in community behavioral health clinics. Behav Ther. 2018;49(4):525–37.

Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40(3):257–65.

Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237–43.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Bullock HL. Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis in: How do systems achieve their goals? the role of implementation in mental health systems improvement [Dissertation]. Hamilton, Ontario: McMaster University; 2019.

Watson DP, Adams EL, Shue S, Coates H, McGuire A, Chesher J, et al. Defining the external implementation context: an integrative systematic literature review. BMC Health Serv Res. 2018;18(1):209.

McKibbon KA, Lokker C, Wilczynski NL, Ciliska D, Dobbins M, Davis DA, et al. A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel? Implement Sci. 2010;5:16.

Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23.

Egan M, Maclean A, Sweeting H, Hunt K. Comparing the effectiveness of using generic and specific search terms in electronic databases to identify health outcomes for a systematic review: a prospective comparative study of literature search method. BMJ Open. 2012;2:3.

Dillman DA, Smyth JD, Christian LM. Internet, mail, and mixed-mode surveys: the tailored design method. Hoboken, NJ: John Wiley & Sons; 2009.

Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation. https://www.covidence.org . Accessed Mar 2019.

Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA, et al. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118.

Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2019.

Henrikson NB, Blasi PR, Dorsey CN, Mettert KD, Nguyen MB, Walsh-Bailey C, et al. Psychometric and pragmatic properties of social risk screening tools: a systematic review. Am J Prev Med. 2019;57(6S1):S13–24.

Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

Lau AS, Brookman-Frazee L. The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implement Sci. 2016;11:1–8.

Ekvall G. Organizational climate for creativity and innovation. European J Work Organizational Psychology. 1996;5(1):105–23.

Lövgren G, Eriksson S, Sandman PO. Effects of an implemented care policy on patient and personnel experiences of care. Scand J Caring Sci. 2002;16(1):3–11.

Dwyer DJ, Ganster DC. The effects of job demands and control on employee attendance and satisfaction. J Organ Behav. 1991;12:595–608.

Condon-Paoloni D, Yeatman HR, Grigonis-Deane E. Health-related claims on food labels in Australia: understanding environmental health officers’ roles and implications for policy. Public Health Nutr. 2015;18(1):81–8.

Patterson MG, West MA, Shackleton VJ, Dawson JF, Lawthom R, Maitlis S, et al. Validating the organizational climate measure: links to managerial practices, productivity and innovation. J Organ Behav. 2005;26:279–408.

Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl. 2012;36(9):621–32.

Beidas RS, Aarons G, Barg F, Evans A, Hadley T, Hoagwood K, et al. Policy to implementation: evidence-based practice in community mental health--study protocol. Implement Sci. 2013;8(1):38.

Eisenberger R, Cummings J, Armeli S, Lynch P. Perceived organizational support, discretionary treatment, and job satisfaction. J Appl Psychol. 1997;82:812–20.

CAS PubMed Google Scholar

Eby L, George K, Brown BL. Going tobacco-free: predictors of clinician reactions and outcomes of the NY state office of alcoholism and substance abuse services tobacco-free regulation. J Subst Abus Treat. 2013;44(3):280–7.

Vogler S, Zimmermann N, de Joncheere K. Policy interventions related to medicines: survey of measures taken in European countries during 2010-2015. Health Policy. 2016;120(12):1363–77.

Wanberg CRB, Banas JT. Predictors and outcomes of openness to change in a reorganizing workplace. J Applied Psychology. 2000;85:132–42.

CAS Google Scholar

Hardy LJ, Wertheim P, Bohan K, Quezada JC, Henley E. A model for evaluating the activities of a coalition-based policy action group: the case of Hermosa Vida. Health Promot Pract. 2013;14(4):514–23.

Gavriilidis G, Östergren P-O. Evaluating a traditional medicine policy in South Africa: phase 1 development of a policy assessment tool. Glob Health Action. 2012;5:17271.

Hongoro C, Rutebemberwa E, Twalo T, Mwendera C, Douglas M, Mukuru M, et al. Analysis of selected policies towards universal health coverage in Uganda: the policy implementation barometer protocol. Archives Public Health. 2018;76:12.

Roeseler A, Solomon M, Beatty C, Sipler AM. The tobacco control network’s policy readiness and stage of change assessment: what the results suggest for moving tobacco control efforts forward at the state and territorial levels. J Public Health Manag Pract. 2016;22(1):9–19.

Brämberg EB, Klinga C, Jensen I, Busch H, Bergström G, Brommels M, et al. Implementation of evidence-based rehabilitation for non-specific back pain and common mental health problems: a process evaluation of a nationwide initiative. BMC Health Serv Res. 2015;15(1):79.

Rütten A, Lüschen G, von Lengerke T, Abel T, Kannas L, Rodríguez Diaz JA, et al. Determinants of health policy impact: comparative results of a European policymaker study. Sozial-Und Praventivmedizin. 2003;48(6):379–91.

Smith SN, Lai Z, Almirall D, Goodrich DE, Abraham KM, Nord KM, et al. Implementing effective policy in a national mental health reengagement program for veterans. J Nerv Ment Dis. 2017;205(2):161–70.

Carasso BS, Lagarde M, Cheelo C, Chansa C, Palmer N. Health worker perspectives on user fee removal in Zambia. Hum Resour Health. 2012;10:40.

Goldsmith REH, C.F. Measuring consumer innovativeness. J Acad Mark Sci. 1991;19(3):209–21.

Webster CA, Caputi P, Perreault M, Doan R, Doutis P, Weaver RG. Elementary classroom teachers’ adoption of physical activity promotion in the context of a statewide policy: an innovation diffusion and socio-ecologic perspective. J Teach Phys Educ. 2013;32(4):419–40.

Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychol Assess. 2010;22(2):356–65.

Gill KJ, Campbell E, Gauthier G, Xenocostas S, Charney D, Macaulay AC. From policy to practice: implementing frontline community health services for substance dependence--study protocol. Implement Sci. 2014;9:108.

Lavinghouze SR, Price AW, Parsons B. The environmental assessment instrument: harnessing the environment for programmatic success. Health Promot Pract. 2009;10(2):176–85.

Bull FC, Milton K, Kahlmeier S. National policy on physical activity: the development of a policy audit tool. J Phys Act Health. 2014;11(2):233–40.

Bull F, Milton K, Kahlmeier S, Arlotti A, Juričan AB, Belander O, et al. Turning the tide: national policy approaches to increasing physical activity in seven European countries. British J Sports Med. 2015;49(11):749–56.

Schneider EC, Smith ML, Ory MG, Altpeter M, Beattie BL, Scheirer MA, et al. State fall prevention coalitions as systems change agents: an emphasis on policy. Health Promot Pract. 2016;17(2):244–53.

Helfrich CD, Savitz LA, Swiger KD, Weiner BJ. Adoption and implementation of mandated diabetes registries by community health centers. Am J Prev Med. 2007;33(1,Suppl):S50-S65.

Donchin M, Shemesh AA, Horowitz P, Daoud N. Implementation of the Healthy Cities’ principles and strategies: an evaluation of the Israel Healthy Cities network. Health Promot Int. 2006;21(4):266–73.

Were MC, Emenyonu N, Achieng M, Shen C, Ssali J, Masaba JP, et al. Evaluating a scalable model for implementing electronic health records in resource-limited settings. J Am Med Inform Assoc. 2010;17(3):237–44.

Konduri N, Sawyer K, Nizova N. User experience analysis of e-TB Manager, a nationwide electronic tuberculosis recording and reporting system in Ukraine. ERJ Open Research. 2017;3:2.

McDonnell E, Probart C. School wellness policies: employee participation in the development process and perceptions of the policies. J Child Nutr Manag. 2008;32:1.

Mersini E, Hyska J, Burazeri G. Evaluation of national food and nutrition policy in Albania. Zdravstveno Varstvo. 2017;56(2):115–23.

Cavagnero E, Daelmans B, Gupta N, Scherpbier R, Shankar A. Assessment of the health system and policy environment as a critical complement to tracking intervention coverage for maternal, newborn, and child health. Lancet. 2008;371 North American Edition(9620):1284-93.

Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22(4):197–209.

Pankratz M, Hallfors D, Cho H. Measuring perceptions of innovation adoption: the diffusion of a federal drug prevention policy. Health Educ Res. 2002;17(3):315–26.

Cook JM, Thompson R, Schnurr PP. Perceived characteristics of intervention scale: development and psychometric properties. Assessment. 2015;22(6):704–14.

Probart C, McDonnell ET, Jomaa L, Fekete V. Lessons from Pennsylvania’s mixed response to federal school wellness law. Health Aff. 2010;29(3):447–53.

Probart C, McDonnell E, Weirich JE, Schilling L, Fekete V. Statewide assessment of local wellness policies in Pennsylvania public school districts. J Am Diet Assoc. 2008;108(9):1497–502.

Rakic S, Novakovic B, Stevic S, Niskanovic J. Introduction of safety and quality standards for private health care providers: a case-study from the Republic of Srpska, Bosnia and Herzegovina. Int J Equity Health. 2018;17(1):92.

Rozema AD, Mathijssen JJP, Jansen MWJ, van Oers JAM. Sustainability of outdoor school ground smoking bans at secondary schools: a mixed-method study. Eur J Pub Health. 2018;28(1):43–9.

Barbero C, Moreland-Russell S, Bach LE, Cyr J. An evaluation of public school district tobacco policies in St. Louis County, Missouri. J Sch Health. 2013;83(8):525–32.

Williams KM, Kirsh S, Aron D, Au D, Helfrich C, Lambert-Kerzner A, et al. Evaluation of the Veterans Health Administration’s specialty care transformational initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577–89.

Spencer E, Walshe K. National quality improvement policies and strategies in European healthcare systems. Quality Safety Health Care. 2009;18(Suppl 1):i22–i7.

Assunta M, Dorotheo EU. SEATCA Tobacco Industry Interference Index: a tool for measuring implementation of WHO Framework Convention on Tobacco Control Article 5.3. Tob Control. 2016;25(3):313–8.

Tummers L. Policy alienation of public professionals: the construct and its measurement. Public Adm Rev. 2012;72(4):516–25.

Tummers L, Bekkers V. Policy implementation, street-level bureaucracy, and the importance of discretion. Public Manag Rev. 2014;16(4):527–47.

Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci. 2008;3:26.

Peters D, Harting J, van Oers H, Schuit J, de Vries N, Stronks K. Manifestations of integrated public health policy in Dutch municipalities. Health Promot Int. 2016;31(2):290–302.

Tosun J, Lang A. Policy integration: mapping the different concepts. Policy Studies. 2017;38(6):553–70.

Tubbing L, Harting J, Stronks K. Unravelling the concept of integrated public health policy: concept mapping with Dutch experts from science, policy, and practice. Health Policy. 2015;119(6):749–59.

Donkin A, Goldblatt P, Allen J, Nathanson V, Marmot M. Global action on the social determinants of health. BMJ Glob Health. 2017;3(Suppl 1):e000603-e.

Baum F, Friel S. Politics, policies and processes: a multidisciplinary and multimethods research programme on policies on the social determinants of health inequity in Australia. BMJ Open. 2017;7(12):e017772-e.

Delany T, Lawless A, Baum F, Popay J, Jones L, McDermott D, et al. Health in All Policies in South Australia: what has supported early implementation? Health Promot Int. 2016;31(4):888–98.

Valaitis R, MacDonald M, Kothari A, O'Mara L, Regan S, Garcia J, et al. Moving towards a new vision: implementation of a public health policy intervention. BMC Public Health. 2016;16:412.

Bennett LM, Gadlin H, Marchand, C. Collaboration team science: a field guide. Bethesda, MD: National Cancer Institute, National Institutes of Health; 2018. Contract No.: NIH Publication No. 18-7660.

Mazumdar M, Messinger S, Finkelstein DM, Goldberg JD, Lindsell CJ, Morton SC, et al. Evaluating academic scientists collaborating in team-based research: a proposed framework. Acad Med. 2015;90(10):1302–8.

Brownson RC, Fielding JE, Green LW. Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annu Rev Public Health. 2018;39:27–53.

Brownson RC, Colditz GA, Proctor EK. Future issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. Second Edition ed. New York: Oxford University Press; 2018.

Thomson K, Hillier-Brown F, Todd A, McNamara C, Huijts T, Bambra C. The effects of public health policies on health inequalities in high-income countries: an umbrella review. BMC Public Health. 2018;18(1):869.

Download references

Acknowledgements

The authors are grateful for the policy expertise and guidance of Alexandra Morshed and the administrative support of Mary Adams, Linda Dix, and Cheryl Valko at the Prevention Research Center, Brown School, Washington University in St. Louis. We thank Lori Siegel, librarian, Brown School, Washington University in St. Louis, for assistance with search terms and procedures. We appreciate the D&I contributions of Enola Proctor and Byron Powell at the Brown School, Washington University in St. Louis, that informed this review. We thank Russell Glasgow, University of Colorado Denver, for guidance on the overall review and pragmatic measure criteria.

This project was funded March 2019 through February 2020 by the Foundation for Barnes-Jewish Hospital, with support from the Washington University in St. Louis Institute of Clinical and Translational Science Pilot Program, NIH/National Center for Advancing Translational Sciences (NCATS) grant UL1 TR002345. The project was also supported by the National Cancer Institute P50CA244431, Cooperative Agreement number U48DP006395-01-00 from the Centers for Disease Control and Prevention, R01MH106510 from the National Institute of Mental Health, and the National Institute of Diabetes and Digestive and Kidney Diseases award number P30DK020579. The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the Foundation for Barnes-Jewish Hospital, Washington University in St. Louis Institute of Clinical and Translational Science, National Institutes of Health, or the Centers for Disease Control and Prevention.

Author information

Authors and affiliations.

Prevention Research Center, Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Peg Allen, Meagan Pilar, Callie Walsh-Bailey, Stephanie Mazzucca, Maura M. Kepper & Ross C. Brownson

School of Social Work, Brigham Young University, 2190 FJSB, Provo, UT, 84602, USA

Cole Hooley

Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Seattle, WA, 98101, USA

Cara C. Lewis, Kayne D. Mettert & Caitlin N. Dorsey

Department of Health Management & Policy, Drexel University Dornsife School of Public Health, Nesbitt Hall, 3215 Market St, Philadelphia, PA, 19104, USA

Jonathan Purtle

Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Ana A. Baumann

Department of Surgery (Division of Public Health Sciences) and Alvin J. Siteman Cancer Center, Washington University School of Medicine, 4921 Parkview Place, Saint Louis, MO, 63110, USA

Ross C. Brownson

You can also search for this author in PubMed Google Scholar

Contributions

Review methodology and quality assessment scale: CCL, KDM, CND. Eligibility criteria: PA, RCB, CND, KDM, SM, MP, JP. Search strings and terms: CH, PA, MP with review by AB, RCB, CND, CCL, MMK, SM, KDM. Framework selection: PA, AB, CH, MP. Abstract screening: PA, CH, MMK, SM, MP. Full-text screening: PA, CH, MP. Pilot extraction: PA, DNC, CH, KDM, SM, MP. Data extraction: MP, CWB. Data aggregation: MP, CWB. Writing: PA, RCB, JP. Editing: RCB, JP, SM, AB, CD, CH, MMK, CCL, KM, MP, CWB. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Peg Allen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors declare they have no conflicting interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: table s1.

. PRISMA checklist. Table S2 . Electronic search terms for databases searched through EBSCO. Table S3 . Electronic search terms for searches conducted through PROQUEST. Table S4: PAPERS Pragmatic rating scales. Table S5 . PAPERS Psychometric rating scales.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Allen, P., Pilar, M., Walsh-Bailey, C. et al. Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implementation Sci 15 , 47 (2020). https://doi.org/10.1186/s13012-020-01007-w

Download citation

Received : 24 March 2020

Accepted : 05 June 2020

Published : 19 June 2020

DOI : https://doi.org/10.1186/s13012-020-01007-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Implementation science
Health policy
Policy implementation
Implementation
Public policy
Psychometric

Implementation Science

ISSN: 1748-5908

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Quality in health care: possibilities and limitations of quantitative research instruments among health care users

Published: 07 October 2011
Volume 47 , pages 1703–1716, ( 2013 )

Cite this article

Mirna Macur 1

3403 Accesses

14 Citations

Explore all metrics

Quality in health care has traditionally been dominated by medical profession, where patients’ opinions were labelled as lay evaluation. Patients’ views and opinions are important because they give us an insight into dimensions of quality that are not evaluated by medical profession and often seem to be more important. In health care quantitative methodology is often used to address these quality dimensions and introduce patients’ views and opinions. There are various benefits using quantitative research instruments, such as a detailed analysis of the importance of various quality dimensions for patients and an analysis of factors influencing patients’ satisfaction. On the other hand serious deficiencies can be tackled too, that are usually dealt with qualitative research instruments, because they go deeper into people’s motives and feelings. However, health care service is specific—it is very important to patients (health is one of the most important values), but their participation in health care service is rather low. They also don’t always say and do what they mean. In such a context combination of quantitative and qualitative research instrument does not give satisfactory answers. The importance of complaints is stressed and rewards for taking them seriously and acting upon them is discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

The Importance of Patient Reported Outcomes in Shaping a Healthcare System

Developing an Evaluation Questionnaire for Healthcare Quality Considering Patient Experience

Development and validation of primary health care quality assessment tool

Bengtsson S.: Quality by users’ influence and involvement in Denmark—the logical development of social services. In: Evers, A., Haverinen, R., Leichsenring, K., Wistow, G. (eds) Developing Quality in Personal Social Services; Concepts, Cases and Comments, pp. 201–214. Ashgate/European Centre, Vienna (1997)

Google Scholar

Bergman B., Klefsjö B.: Quality; from Customer Needs to Customer Satisfaction. McGraw Hill Book Company, London (1994)

Berry L.L., Parasuraman A.: Marketing Services Competing Through Quality. The Free Press, A Division of Macmillan, Inc., New York (1991)

Dowing M.: Measuring quality in personal social services. In: Evers, A., Haverinen, R., Leichsenring, K., Wistow, G. (eds) Developing Quality in Personal Social Services; Concepts, Cases and Comments, pp. 95–108. Ashgate/European Centre, Vienna (1997)

Družina umrlega 12-letnika zaradi malomarnosti ovadila zdravnike. Slovenija—zdravje 26 Nov 2008. http://www.siol.net/slovenija/zdravje/2008/11/zaradi_malomarnosti_ovadili_zdravnike.aspx (2011). Accessed 22 May 2011

Fras Z., Pajntar M.: Kako merimo kakovost zdravstvenega varstva. Zdravniški vestnik 71 (12), 737–741 (2002)

Göb R., McCollin C., Ramalhoto M.F.: Ordinal methodology in the analysis of Likert scales. Qual. Quant. 41 , 601–626 (2007)

Article Google Scholar

Goričan, A.: Zadovoljstvo uporabnikov zdravstvenih storitev z delom zdravstvenega osebja: študija primera: magistrska naloga. Univerza na primorskem, Fakulteta za management v Kopru (2009)

Hirschman A.O.: Exit, Voice and Loyalty. Response to Decline in Firms, Organisations and States. Howard University Press, Cambridge (1970)

Kersnik J.: Zadovoljstvo bolnikom z zdravnikom družinske medicine. Doktorska disertacija, Medicinska fakulteta, Ljubljana (2011)

Kersnik, J.: Kakovost v sistemu zdravstvenega varstva v Republiki Sloveniji—pregled stanja. Ministrtvo za zdravje. Urad SZO. http://www.drmed.org/projekt_kakovost/Porocilo_za_WHO_3.doc (2011). Accessed 5 May 2011

Kumar R.: Reserch Methodology. A Step by Step Guide for Beginners. Sage Publications, London (2005)

Kutin B., Dovjak Plešej M., Macur M., Zore L., But D.: Pritožni postopki kot mehanizem uveljavljanja pravic zavarovanih oseb v institucijah zdravstvenega varstva. Research report. Zveza Potrošnikov Slovenije, Ljubljana (2001)

Lariot N.: Krik (Grasset et Fasquelle). Prešernova družba, Ljubljana (1978)

Macur M.: Vpliv privatizacije na kakovost zdravstvenih storitev—evalvacija z vidika uporabnika. Doktorska disertacija, Fakulteta za družbene vede, Ljubljana (2000)

Oppen M.: Towards a new client orientation through continuous improvement. In: Evers, A., Haverinen, R., Leichsenring, K., Wistow, G. (eds) Developing Quality in Personal Social Services; Concepts, Cases and Comments, pp. 109–126. Ashgate/European Centre, Vienna (1997)

Pestoff, V.A.: Citizens as co-producers of social services in Europe. From the welfare state to the welfare mix. Stockholm University, School of Business, Research Report (1995)

Pfeffer, N., Coote, A.: Is quality good for you?; A critical review of quality assurance in welfare services. Social Policy Paper No. 5, London (1991)

Potočnik V.: Trženje storitev. Gospodarski vestnik, Ljubljana (2000)

Remic M.: Porabnikovo vrednotenje zdravstvenih storitev. Diplomsko delo, Ekonomska fakulteta, Ljubljana (2003)

Rus V., Toš N.: Vrednote Slovencev in Evropejcev: analiza vrednotnih orientacij Slovencev ob koncu stoletja. Fakulteta za družbene vede, Ljubljana (2005)

Sale J.E.M., Lohfeld L.H., Brazil K.: Revisiting the quantitative—qualitative debate: implications for mixed-methods research. Qual. Quant. 36 , 43–53 (2002)

Švab, I., Kersnik, J., Berger, T., Rotar-Pavlič, D., Vegnuti, M., Poplas-Sušič, T., Vatovec-Progar, I.: Kakovost dela v osnovni zdravstveni dejavnosti v letu 1997; letno poročilo. Inštitut za varovanje zdravja Republike Slovenije, Ljubljana (1997)

Toš, N., et al.: Slovensko javno mnenje 2001/3: Raziskava o zdravju in zdravstvu IV. in Raziskava o obrambi in varnosti [datoteka podatkov]. Ljubljana: Univerza v Ljubljani, Fakulteta za družbene vede, Center za raziskovanje javnega mnenja in množičnih komunikacij [izdelava], 2001. Ljubljana: Univerza v Ljubljani, Fakulteta za družbene vede, Arhiv družboslovnih podatkov [distribucija] (2002)

Uhan S.: Prava in neprava mnenja. Znanstvena knjiznica, Fakulteta za druzbene vede, Ljubljana (1998)

Vidovič D.: Zadovoljstvo bolnikov kot merilo kakovosti v splošni medicini. Prešernova naloga. Medicinska fakulteta, Ljubljana (1998)

WHO: Quality/patient safety. http://www.wpro.who.int/health_topics/quality_patient_safety/ (2011). Accessed 17 May 2011

Download references

Author information

Authors and affiliations.

School of Advanced Social Studies, Gregorčičeva ulica 19, 5000, Nova Gorica, Slovenia

Mirna Macur

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mirna Macur .

Rights and permissions

Reprints and permissions

About this article

Macur, M. Quality in health care: possibilities and limitations of quantitative research instruments among health care users. Qual Quant 47 , 1703–1716 (2013). https://doi.org/10.1007/s11135-011-9621-z

Download citation

Published : 07 October 2011

Issue Date : April 2013

DOI : https://doi.org/10.1007/s11135-011-9621-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quantitative research instruments
Quality of health care services
Patients’ satisfaction
Find a journal
Publish with us
Track your research

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
Current issue
BMJ Journals

Main results: Of 98 prospective HIA studies, 17 reported quantitative estimates of change in exposure to determinants, and 16 gave quantified health outcomes. Eleven (categories of) determinants were quantified up to the level of health outcomes. Methods for exposure impact assessment were: estimation on the basis of routine data and measurements, and various kinds of modelling of traffic related and environmental factors, supplemented with experts’ estimates and author’s assumptions. Some studies used estimates from other documents pertaining to the policy. For the calculation of health outcomes, variants of epidemiological and toxicological risk assessment were used, in some cases in mathematical models.

Conclusions: Quantification is comparatively rare in HIA. Methods are available in the areas of environmental health and, to a lesser extent, traffic accidents, infectious diseases, and behavioural factors. The methods are diverse and their reliability and validity are uncertain. Research and development in the following areas could benefit quantitative HIA: methods to quantify the effect of socioeconomic and behavioural determinants; user friendly simulation models; the use of summary measures of public health, expert opinion and scenario building; and empirical research into validity and reliability.

health impact assessment
quantitative methods
determinants

https://doi.org/10.1136/jech.2004.026039

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Funding: this work was sponsored by ZonMw, the Netherlands Institute of Health Sciences and the Foundation Vereniging Trustfonds Erasmus Universiteit Rotterdam.

Competing interests: none declared.

Linked Articles

In this issue Common sense, the least common sense? Carlos Alvarez-Dardet John R Ashton Journal of Epidemiology & Community Health 2005; 59 341-341 Published Online First: 14 Apr 2005.

Read the full text or download the PDF:

Research article
Open access
Published: 19 May 2017

Influences on patient satisfaction in healthcare centers: a semi-quantitative study over 5 years

Ruth D. Thornton 1 ,
Nicole Nurse 2 ,
Laura Snavely 3 ,
Stacey Hackett-Zahler 4 ,
Kenice Frank 5 &
Robert A. DiTomasso 1

BMC Health Services Research volume 17 , Article number: 361 ( 2017 ) Cite this article

25k Accesses

29 Citations

6 Altmetric

Metrics details

Knowledge of ambulatory patients’ satisfaction with clinic visits help improve communication and delivery of healthcare. The goal was to examine patient satisfaction in a primary care setting, identify how selected patient and physician setting and characteristics affected satisfaction, and determine if feedback provided to medical directors over time impacted patient satisfaction.

A three-phase, semi-quantitative analysis was performed using anonymous, validated patient satisfaction surveys collected from 889 ambulatory outpatients in 6 healthcare centers over 5-years. Patients’ responses to 21 questions were analyzed by principal components varimax rotated factor analysis. Three classifiable components emerged: Satisfaction with Physician, Availability/Convenience, and Orderly/Time. To study the effects of several independent variables (location of clinics, patients’ and physicians’ age, education level and duration at the clinic), data were subjected to multivariate analysis of variance (MANOVA)..

Changes in the healthcare centers over time were not significantly related to patient satisfaction. However, location of the center did affect satisfaction. Urban patients were more satisfied with their physicians than rural, and inner city patients were less satisfied than urban or rural on Availability/Convenience and less satisfied than urban patients on Orderly/Time.

How long a patient attended a center most affected satisfaction, with patients attending >10 years more satisfied in all three components than those attending <1–5 years. Level of education affected patients’ satisfaction only in the component Orderly/Time; patients without a high school education were significantly less satisfied than those with more. Patients in their 40′s were significantly less satisfied in Availability/Convenience than those >60 years old.

Patients were significantly more satisfied with their 30–40 year-old physicians compared with those over 60. On Orderly/Time, patients were more satisfied with physicians who were in their 50′s than physicians >60.

Conclusions

Improvement in patient satisfaction includes a need for immediate, specific feedback. Although Medical Directors received feedback yearly, we found no significant changes in patient satisfaction over time. Our results suggest that, to increase satisfaction, patients with lower education, those who are sicker, and those who are new to the center likely would benefit from additional high quality interactions with their physicians.

Peer Review reports

Patient satisfaction surveys are often used to understand patients’ concerns and determine areas for improvement, including improving communication between physicians and patients. Survey results document progress and allow physicians and staff to maintain high standards. Although results of patient satisfaction surveys are used by payer systems to profile individual physicians and guide physician compensation, one study showed that < 25% of primary care physicians found profiles useful for improving patient care and fewer used the profiles to change [ 1 ]. Improvements are more likely to occur if staff receives more immediate feedback [ 2 ].

Data collection methods play a role in outcomes. On-site surveys provide an immediate outlet for patients who are experiencing problems, although higher ratings for on-site surveys may also relate directly to doctor-patient communication. Surveys administered later after a clinic visit may yield lower ratings, possibly due to the course of treatment [ 2 , 3 ].

Many factors influence patient satisfaction. Patient demographics such as age, gender, income, socioeconomic and general health status impact patients’ responses [ 3 , 4 ]. Characteristics of the medical provider, including demographics and experience, also affect their interactions with patients [ 5 , 6 , 7 , 8 , 9 ]. Other factors include the type of setting the patient is in [ 10 ] and the amount of time patients had to wait [ 11 ]. However, Anderson found that complaints about wait time can be moderated by time spent with the physician [ 12 ].

Physician characteristics extend beyond the obvious. Physician-patient concordance in race, gender or age may be important in patient satisfaction, but many other factors such as primary language, parental status, sexual orientation, values, beliefs, or communication style may be associated [ 13 , 14 ]. How long the patient has been with this physician and the degree to which the physicians’ communication is patient-centered are significant [ 13 ]. A physician’s experience plays a role, with lowest patient satisfaction with first-year residents; interestingly, residents with some more experience attained similar satisfaction ratings to those of the faculty attendings, suggesting that the requisite skills are acquired during the first year of training [ 7 ].

Whether to administer patient satisfaction surveys depends on the overall goals of the medical facility and on physician buy-in to change [ 1 , 15 ]. The views of the medical director and administrator are key components as to whether the surveys are taken seriously and acted upon by physicians [ 16 ]. Patient satisfaction can become a success criterion of the center when physicians and staff meet regularly to discuss improvements in a context of cooperation and mutual support.

We initiated this study of patient satisfaction to help physicians better understand their patients at the healthcare centers (HCCs) of a not-for-profit medical school’s outpatient primary care centers on the east coast. Physicians were provided raw data and results of open-ended questions very soon after each year’s study. However, we decided to statistically analyze the overall data in order to understand where patients were most and least satisfied and what influenced their satisfaction. Our goal was to provide information which could help focus physician directors’ changes to improve patient satisfaction.

The research was under the auspices of a medical college (Philadelphia College of Osteopathic Medicine, PCOM) which owns and operates five outpatient HCCs, four of which are located within the city limits of Philadelphia and the fifth HCC located in a rural area. [ 17 ] Two within Philadelphia are considered urban, while two are in the inner city [ 18 ]. An additional nonaffiliated, inner city HCC located within Philadelphia was also used in the research. We considered the nonaffiliated HCC as a control, but expected it to likely agree with data from the affiliated inner city HCCs. The quantity of surveys administered are listed in Table 1 .

This research arose from a need to quickly and inexpensively conduct patient satisfaction surveys in the Healthcare Centers, incorporating a research component involving graduate students interested in health related careers. Surveys were administered to patients at the five HCCs. Patient questions were adapted from the validated DiTomasso-Willard Patient Satisfaction Questionnaire [ 19 ] (questions are listed in Table 2 ). Demographic information and responses to open-ended questions were also collected. In 2005 (year 1), 2007 (year 2), and 2010 (year 3), students in a master’s program at the medical school approached patients in the waiting areas at each HCC asking them to complete a survey. Patients could take the surveys with them into the examination room, but they returned the survey before leaving the HCC. If requested, the student helped a patient read the questions.

Each surveying period was conducted over an approximately one month of time. Students varied their sampling by time of day and day of week. Therefore, the sample was comprised of a random representation of patients attending each HCC during each one-month period of surveying. The students approached anyone who was in the waiting room during sampling times, but patients were free to refuse if they wished. The goal was to obtain approximately 10% of the average number of patients seen by each HCC in a month.

The protocol (Protocol #H05-022X) was approved by the Institutional Review Board (IRB) of PCOM that determined it to be exempt from informed consent requirements under 45 CFR 46.101(b)(2)--survey research in which the responses will be recorded in such a manner that the human subjects cannot be identified, directly or through identifiers linked to the subjects (e.g., name, Social Security number). Further, no master list existed linking such identifiers to the subjects. Approximately 5–15% of the average numbers of patients coming to each HCC in a month were surveyed. Inclusion criteria included patients willing to respond, patient age of at least 18 years, and patients who spoke English. Patients were assured the questionnaire was confidential without any identifying information, the results would be presented in aggregate form, and that their responses would not affect their specific care at the HCC. In order to maintain anonymity, a patient’s medical status was not requested, although in retrospect, it may have been helpful. From observation, students reported that those with acute medical issues were less inclined to participate. Although an absolute count was not performed, students who administered surveys consistently estimated that only about 5% of the patients in the waiting room refused to participate.

Survey results were entered into IBM’s Statistical Package for the Social Sciences (SPSS 18.0) for analysis. Missing data were filled in using Linear Interpolation, and any negative questions were transformed to the positive on the Likert scale, so that, for all questions, 5 (strongly agree) meant “most satisfied.” All 21 survey statements were subjected to a principal components varimax rotated factor analysis according to Kaiser’s criterion [ 20 ] which ultimately allowed for a reduction of statements into three classifiable components, Satisfaction with Physician, Availability/Convenience, and Orderly/Time (Table 2 ).

Following each survey period, the data were analyzed in SPSS to collapse the questions into three classifiable components/categories. These three categories did not vary during the 3 data collection periods. After each survey period, study staff attended face-to-face meetings with Medical Directors of each healthcare center, the Dean of the Medical School, and the Chair of Family Medicine to present the results. HCC staff were provided with mean scores for each question for their HCC compared with a composite of all HCC’s. They also received the data collapsed into the three categories for their HCC compared with a composite of all HCC’s, but without statistical analysis.

For analysis of the composite data, multivariate analysis of variance (MANOVA) was performed for groups of data, using post hoc Tukey to distinguish specific significance between groups. Independent t -test was used for gender analysis, and Chi square analysis was done to compare the observed gender data from patients who completed surveys with patient demographics of each HCC. See Additional Data for more specific information.

In using factor analysis, it is common practice to require 10 subjects per number of items. In the present case, this criterion was far exceeded. For the separate MANOVA analyses using 3 dependent variables, setting power at 95% for a medium effect size at the 0.05 level of significance comparing 2 levels (male vs. female) of the independent variable, 3 levels (3 locations) and 5 levels (physician age groups), the required number of subjects was 280, 171, and 145 respectively. In all cases there was sufficient power.

Surveys were administered to a total of 889 patients who visited one of the HCCs for treatment (Table 1 ). These numbers represented between 5–15% of the average number of patients seen monthly in the affiliated HCCs, and comparable numbers of surveys were obtained from the much larger, non-affiliated HCC.

Applying principal components varimax rotated factor analysis to the survey responses resulted in groups of identifiable questions that constituted factors (Rotated component matrix for all questions is shown on Table 2 ). Three classifiable factors, Satisfaction with Physician, Availability/Convenience, and Orderly/Time, emerged from the analysis and are used throughout this research. Two questions (Q6 and Q7) were not included as the items did not load on any of the factors (Table 2 ). Using the survey questions that constituted each factor (Table 2 ), the three factors have the following characteristics: Satisfaction with Physician involves being satisfied with the quality of medical care received, as well as the physician spending enough time with the patient. Availability/Convenience involves being satisfied with the staff and their helpfulness in making appointments, whether in person or by phone. Orderly/Time has to do with patients’ time being respected, and interactions with staff and physicians being clear and to the point, avoiding confusion.

Overall, patients were quite satisfied with their HCCs, as evidenced by overall mean scores greater than 3.89 on a Likert scale of 1–5 (see Additional file 1 : Table S3A). Mean scores were highest in Satisfaction with Physician (4.27 ± 0.65), while Availability/Convenience (3.92 ± 0.69) and Orderly/Time (3.89 ± 0.66) were somewhat lower. Even so, a score of 3.9 represents the top 20–25% of satisfaction. The open-ended responses emphasized the importance of patients’ satisfaction with their physician, even if patients were somewhat less satisfied with other aspects of their visit (see Additional file 2 : Table S6).

The goal of this research was to identify areas found to be statistically significant. More complete data can be found in the Additional files 1 , 2 , 3 ,and 4 . Based on MANOVA, there was no significance over time in any of the three categories (see Additional file 1 : Table S3B). This points to a consistency over time in the operations and functioning of these HCC’s.

The following areas were found to be statistically significant by MANOVA:

Analyzing satisfaction in inner city, urban and rural HCCs (Fig. 1 ), significance was observed in the following area.: Patients in inner city HCCs were less satisfied than those in urban or rural HCC’s on Availability/Convenience, and those in inner city HCCs were less satisfied than urban patients in the area of Orderly/Time. Urban patients were more satisfied with their Physician than were rural patients while inner city patients’ satisfaction with their Physician was not significantly different from the other localities (See Additional file 1 : Table S3C, for more detail).

Satisfaction by location (Inner City, Urban and Rural). Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

When individual HCCs were analyzed (Fig. 2 ), one urban HCC (#5) had significantly higher satisfaction with their Physician than the other urban HCC (#2) or one inner city HCC (#6). The other urban HCC (#2) had more satisfaction in the category of Orderly/Time than two of the three inner city HCCs (#3 and #6). Two inner city HCCs (#1 and #6) had significantly lower satisfaction in the category of Availability/Convenience than the rural HCC (#4). (See Additional file 1 : Table S3D, for details.)

Satisfaction by individual HCCs. Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

Patients’ demographics appear to play a role in the level of satisfaction. Patients over 60 years old were more satisfied with the Availability/Convenience of the HCC than patients who were in their 40′s (Fig. 3 ). Those with more education (in the range from graduating high school through graduate school) were more satisfied with the Orderly/Time category than those with less than a high school diploma (Fig. 4 ). Finally, patients who had been with their HCC for longer periods of time were more satisfied than those who had been there less than 5 years in all three categories of satisfaction with Physician, Availability/Convenience, and Orderly/Time (Fig. 5 ) (See Additional file 2 : Table S4C, for details).

The effect of patient age on satisfaction. Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

The effect of patient education on satisfaction. Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

The effect of length of time at a HCC on satisfaction. Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

Physicians in these centers tended to longevity in their positions. Patients were more satisfied with their Physicians who were in their 30′s and 40′s than with physicians in their 50′s (Fig. 6 ). Also, physicians in their 50′s were perceived to be more Available than those in their 60′s. Patients rated male physicians as more Available than female physicians, and in the Inner City HCCs, patients rated their Caucasian physicians higher on Availability than African American physicians (see Additional file 3 : Table S5B and C).

The effect of physician’s age on satisfaction. Lines/Brackets indicate comparisons by color that were significantly different in each of the categories

Open ended responses were overall very positive, with the exception of the rural HCC4 during year 1. After personnel replacements at this HCC, more positive responses were also seen there. Wait times were seen as a problem in some HCCs, particularly in the inner city centers. The majority of patients were very satisfied with the convenience of their HCC (See Additional file 4 : Table S6).

In the examination of changes over time, patient satisfaction at the HCCs in the study remained overall quite high in all three categories of Satisfaction with Physician, Availability/Convenience, and Orderly/Time. Meanwhile, notable changes at the affiliated centers during this time period included a new telephone system installed between years 1–2, major renovations of one of the centers in year 2, and installation of a system of Electronic Medical Records (EMR) in all affiliated HCCs between years 2 and 3; the non-affiliated HCC #6 also introduced EMR prior to year 3. While telephone changes would likely affect staff-patient interactions, instituting EMR represented a major change in the physician-patient interactions, with the addition of computers to each examination room. We were surprised that these seemingly “major” changes did not significantly affect the satisfaction levels over this time period. De Leon et al. found generally higher patient satisfaction with a center after EMR were introduced [ 21 ], while we found no significant differences after EMR was installed.

Results of the patient satisfaction surveys were presented to Medical Directors and staff in a timely manner after each survey period, but without statistical analysis. From the initial data given to each HCC, medical staff could compare their mean results with a composite mean result for all the centers. However, they did not have access to comparisons of individual HCCs (see Additional file 1 , Table S3D). Nor did they have access to figures such as Fig. 2 , comparing individual HCCs. It is not surprising that each HCC is unique. An example is HCC 5 with a significantly higher level of satisfaction with Physician compared with two other HCCs, one urban and inner city (Fig. 2 , and Additional file 1 : Table S3D). This merits more in-depth analysis of the physician practices at this outstanding urban HCC as a positive example for others.

We projected that HCCs sharing similar locations (inner city, urban or rural) would be more alike and this proved to be the case. In the components of Availability/Convenience and Orderly/Time, patients in the inner city HCCs were less satisfied than those in urban or rural settings, consistent with findings of the individual HCCs. There could be several reasons for differences between inner city and other HCCs. Fan et al. found that functional status (disease severity, physical limitation) was only weakly associated with general satisfaction, while education, coping skills and disease perception were more important to patient’s satisfaction [ 4 ]. Patients in the inner city may be sicker due to overall inadequate health knowledge or reluctance to visit a doctor, possibly due to lack of insurance. These findings suggest that physician-patient interactions with the goal of improved disease understanding might help as much as actual improvement in health. We did not ask for the health status of individual patients in our survey, so we can only guess the health status of patients at different locations.

Comparing locations (Fig. 1 ) with individual HCCs (Fig. 2 ) reveals the sources of these differences. For example, in Fig. 1 , inner city patients were statistically less satisfied in the component of Orderly/Time than were patients in urban settings. Fig. 2 shows that the differences were primarily with one urban HCC #2 (but not with urban HCC #5), compared only with 2 inner city HCC #3 and #6 (but not with inner city HCC #1). So generalizations require examining the individual HCCs as well.

Education level of the patient can also be reflective of location. Approximately 80% of inner city respondents reported having high school education or less, similar to rural patients (76%), while only 58% of urban patients had a high school education or less (data not shown). Other issues facing patients, such as availability of public transportation, may be more of an obstacle in the inner city than in either urban or rural settings. While public transportation is also not widely available in rural settings, it is likely most patients have access to a vehicle. Inner city respondents also were less satisfied in the component of Orderly/Time than respondents in urban settings, and this is confirmed in the open-ended questions (see Additional file 4 : Table S6) where a larger number of patients specifically mentioned the wait time as a problem in the inner city HCCs than in the urban or rural HCCs. Although we wondered if dissatisfaction with wait time could be directly attributable to student participation in the examination room, that seems not to be the case, as a very small percent of responders mentioned students in the open-ended questions and half of those were positive. Mol et al. found that patients generally felt neutral or positive about the presence of students, and in that study, between 83 and 98% of patients consented to student participation [ 22 ].

Our only finding of differences associated with education level in satisfaction were in the area of Orderly/Time; patients with less than a high school education were less satisfied in the component of Orderly/Time than any other group. This could be due to their inability to understand the medical parlance or the protocols involved in their care. However, one study also found that the converse-a physician’s satisfaction with a patient-was associated with their patients’ higher education level [ 23 ], suggesting that the responsibility may be reciprocal between the physician and the patient.

Another patient demographic of age can also contribute to patient satisfaction. Our finding that patients over 60 years old had a higher degree of satisfaction in Availability/Convenience is not surprising. This finding agrees with Jackson who reported that patients over 65 years old and with higher functional status were more satisfied [ 3 ]. Peck found that physicians were more likely to have patient-centered encounters with patients over age 65, which in turn meant that older patients were more satisfied [ 24 ]. Although there was no impact of patients’ gender on level of satisfaction, we did find that, in general, more female patients agreed to fill out the surveys than were actually represented as patients in the HCCs. Not surprisingly, the most significant differences were found in the length of time a patient had been attending their HCC. This is undoubtedly a self-selection, where either the physician or the location suits the patient who continues to visit that center. Pelletier calls this “sampling bias,” citing that “those who stay with a program…may be those who are most satisfied” [ 23 ]. Another explanation is through “visit continuity,” where respondents rated the quality of physician-patient interaction as being more important during the early stages of continuity or when the patient reported worse self-rated health [ 25 ]. This suggests that physicians who focus on those newer patients or sicker patients who would benefit the most from additional interactions may have the most positive results over time.

Demographics of the physician may also be important to patient satisfaction. The physicians at the affiliated HCCs were all osteopathic (DO) physicians, who self-reported that they used Osteopathic Manipulative Treatment at their clinics about 20% of the time. In the open-ended questions, some patients did express a preference for DO physicians. On age of physicians, it appears that more patients prefer a physician younger than 50 years old in the component of Satisfaction with Physician, but in Orderly/Time, they prefer a physician in their 50′s rather than in their 60′s. We speculate that physicians in their 50′s are likely to be at the pinnacle of their profession, although other considerations may also be important, such as humor or degree of connection that the patient perceives with that physician. In the variable of Orderly/Time, it is possible that physicians in their 50′s may be more efficient, having a well-run visit, while the slower, possibly more thorough pace of older physicians may not be as appreciated.

The statistical significances found in this data enhance the details which were presented to the medical directors after each surveying period and provide additional measures of patient satisfaction. Presenting the data to medical directors in figure form rather than as graphs is likely to enhance understanding. Finally, presenting the data of each individual HCC rather than as a composite may help medical directors to see the larger picture.

The present study has several limitations: In retrospect from patients’ written responses, an additional choice under the education demographic would have better captured any additional education received, such as technical certificates or Associate degrees. Also, the severity of the patient’s medical condition should have been noted, as this has been shown to influence patient satisfaction [ 4 ]. In addition, the questions that fell under the component Orderly/Time in the factor analysis fortuitously contained all questions which had been originally stated in the negative and then were reversed for analysis. Finally, presentation of the data to the medical directors in a timely fashion could be improved by presenting figures in addition to tables, and showing results of each individual HCC.

This study was designed to provide feedback to Medical Directors on patient satisfaction in their HCCs. Our findings point to a consistency in the operations and functioning of these HCCs over time, even when renovations or installation of EMR were performed. Differences in locality (inner city, urban, rural) were found, as well as differences in satisfaction by patient demographics (age, education level, length of time with a HCC) and by physician demographics (age, gender). However, uniqueness of individual HCCs contributes to these differences. Physicians from each HCC regularly meet together, and they can use these meetings to help better understand and build on their strengths and individuality. Results of this study can be used to increase satisfaction if physicians help their patients benefit from their services and increase their satisfaction. Particularly, physicians can concentrate on providing additional high-quality interactions for patients with less education, those who are sicker, and those who are new to the HCC.

Abbreviations

Doctor of Osteopathic Medicine

Electronic Medical Records

Healthcare Centers

Institutional Review Board

Multivariate analysis of variants

Philadelphia College of Osteopathic Medicine

Statistical Package for the Social Sciences

Rider EA, Perrin JM. Performance profiles: the influence of patient satisfaction data on physicians’ practice. Pediatrics. 2002;109(5):752–7.

Article PubMed Google Scholar

Burroughs TE, Waterman BM, Gilin D, Adams D, McCollegan J, Cira J. Do on-site patient satisfaction surveys bias results? Jt Comm J Qual Pt Safety. 2005;31(3):158–66.

Google Scholar

Jackson JL, Chamberlin J, Kroenke K. Predictors of patient satisfaction. Soc Sci Med. 2001;52(4):609–20.

Article CAS PubMed Google Scholar

Fan VS, Reiber GE, Diehr P, Burman M, McDonell MB, Fihn SD. Functional status and patient satisfaction: a comparison of ischemic heart disease, obstructive lung disease, and diabetes mellitus. J Gen Intern Med. 2005;20(5):452–9.

Article PubMed PubMed Central Google Scholar

Fan VS, Burman M, McDonell MB, Fihn SD. Continuity of care and other determinants of patient satisfaction with primary care. J Gen Intern Med. 2005;20(3):226–33.

Fairchild DG, McLoughlin KS, Gharib S, Horsky J, Portnow M, Richter J, Gagliano N, Bates DW. Productivity, quality, and patient satisfaction: comparison of part-time and full-time primary care physicians. J Gen Intern Med. 2001;16(10):663–7.

Article CAS PubMed PubMed Central Google Scholar

Monk SM, Nanagas MT, Fitch JL, Stolfi A, Pickoff AS. Comparison of resident and faculty patient satisfaction surveys in a pediatric ambulatory clinic. Teach Learn Med. 2006;18(4):343–7.

Rodriguez HP, von Glahn T, Chang H, Rogers WH, Safran DG. Measuring patients’ experiences with individual specialist physicians and their practices. Am J Med Qual. 2009;24(1):35–44.

Ware Jr JE, Davies AR. Behavioral consequences of consumer dissatisfaction with medical care. Eval Prog Plan. 1983;6(3–4):291–7.

Article Google Scholar

Perneger TV, Etter JF, Raetzo MA, Schaller P, Stalder H. Comparison of patient satisfaction with ambulatory visits in competing health care delivery settings in Geneva, Switzerland. J Epidemiol Comm Health. 1996;50(4):463–8.

Article CAS Google Scholar

Feddock CA, Hoellein AR, Griffith 3rd CH, Wilson JF, Bowerman JL, Becker NS, Caudill TS. Can physicians improve patient satisfaction with long waiting times? Eval Health Prof. 2005;28(1):40–52.

Anderson RT, Camacho FT, Balkrishnan R. Willing to wait?: the influence of patient wait time on satisfaction with primary care. BMC Health Serv Res. 2007;7:31.

Street Jr RL, O’Malley KJ, Cooper LA, Haidet P. Understanding concordance in patient-physician relationships: personal and ethnic dimensions of shared identity. Ann Fam Med. 2008;6(3):198–205.

Thornton RLJ, Powe NR, Roter D, Cooper LA. Patient-physician social concordance, medical visit communication and patients’ perceptions of health care quality. Pat Educ Couns. 2011;85:e201–8.

Gooding TD, Newcomb L, Mertens K. Patient-centered measurement at an academic medical center. Jt Comm J Qual Improv. 1999;25(7):343–51.

CAS PubMed Google Scholar

Conner DR. Managing at the Speed of Change. 1st ed. Ed Anonymous New York: Villard Books; 1993. p. 105–24.

US Dept of Agriculture. Econ Res Service. Rural Classification. http://www.ers.usda.gov/topics/rural-economy-population/rural-classifications/what-is-rural.aspx . Accessed 2016.

Merriam-Webster Dictionary. Urban & Inner City. http://www.merriam-webster.com/dictionary . Accessed 2016.

DiTomasso RA, Willard M. The development of a patient satisfaction questionnaire in the ambulatory setting. Fam Med. 1991;23(2):127–31.

Field A. Discovering statistics using SPSS. 2nd ed. 2005.

De Leon DF, Silfen SL, Wang JJ, Kamara TS, Wu WY, Shih SC. Patient experiences at primary care practices using electronic health records. J Med Pract Manage. 2012;28(3):169–76.

PubMed Google Scholar

Mol SSL, Peelen JH, Kuyvenhoven MM. Patients’ view on student participation in general practice consultations: A comprehensive review. Med Teacher. 2011;33:e397–400.

Pelletier M. Client satisfaction surveys: variables to watch out for. Dimens Health Serv. 1985;62(1):37–9.

Peck BM. Age-related differences in doctor-patient interaction and patient satisfaction. Curr Gerontol Geriatr Res. 2011;2011:137492–501.

Rodriguez HP, Rogers WH, Marshall RE, Safran DG. The effects of primary care physician visit continuity on patients’ experiences with care. J Gen Intern Med. 2007;22(6):787–93.

Download references

Acknowledgements

The authors wish to acknowledge Audrey Rossowski, M.S. for collecting surveys at the healthcare centers and Barbara A. Mitchell, Ph.D. for helpful editing. We also sincerely thank PCOM Kenneth Veit, DO, MBA, FACOFP, Provost, Senior Vice President for Academic Affairs and Dean. We also thank PCOM Family Medicine Chair and Director Harry Morris, DO; PCOM Healthcare Center Physicians and Medical Directors, Oliver Bullock, DO., Michael Becker, DO, Izola David, DO, Larry Finklestein, DO, David Kuo, DO, Marta Motel, DO, Barbara Williams-Page, DO, and David Wood, DO; and A. Scott McNeal, DO, Fairmount Primary Care Center, for allowing our participation.

An abstract of this research was presented by Kenice Frank, M.S., at the Student National Medical Association’s National Meeting, May, 2012, Atlanta, GA.

No funding sources were used for this research project.

Availability of data and materials

The SPSS data and subsequent analysis is available from RDT and the corresponding author, RAD, on reasonable request. A large part of the data is already shown in Additional files 1 , 2 , 3 , and 4 , associated with this publication.

Authors’ contributions

RDT conceived of the research, wrote the paper, and, with the help of RAD, analyzed the data. NN received her M.S. degree from PCOM for her research on the first year (2005) of the study. She gathered the data, analyzed it, and presented a thesis, “A measure of patient satisfaction of PCOM healthcare centers based upon geographic settings.” LS received her M.S. degree from PCOM for her research on the first year (2005) of the study. She gathered the data, analyzed it, and presented a thesis, “The effects of demographic factors on patient satisfaction in an ambulatory setting.” SH-Z received her M.S. degree from PCOM for her research on the second year (2007) of the study. She gathered the data for 2007, analyzed the 2007 data and compared it with 2005 data, and presented a thesis, “Examining improvement levels in measured patient satisfaction in an academic primary care setting.” KF gathered the data on the third year (2010) of the study. She analyzed the 2010 data, compared it with the 2005 and 2007 data, and presented a poster on the study at the Student National Medical Association in 2011. RAD gave guidance to the entire study, providing the original survey questions [ 17 ], analyzing statistically all of the data, serving on each M.S. student’s thesis committee, and editing the paper. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

The protocol was approved by the Institutional Review Board (IRB) at Philadelphia College of Osteopathic Medicine that determined it to be exempt from informed consent requirements under 45 CFR 46.101(b)(2)-survey research in which the responses will be recorded in such a manner that the human subjects cannot be identified, directly or through identifiers linked to the subjects (e.g., name, Social Security number). Further, no master list existed linking such identifiers to the subjects. Inclusion criteria included patients willing to respond, patient age of at least 18 years, and patients who spoke English. Patients were assured the questionnaire was confidential without any identifying information, the results would be presented in aggregate form, and that their responses would not affect their specific care at the HCC.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and affiliations.

Department of Psychology, Philadelphia College of Osteopathic Medicine, 4170 City Ave., Philadelphia, PA, 19131, USA

Ruth D. Thornton & Robert A. DiTomasso

North Philadelphia Health System/St. Joseph’s Hospital, Philadelphia, PA, USA

Nicole Nurse

Geisinger Medical Center, Danville, PA, USA

Laura Snavely

Maria Fareri Children’s Hospital, Westchester, NY, USA

Stacey Hackett-Zahler

College of Podiatric Medicine & Surgery, Des Moines University, Des Moines, IO, USA

Kenice Frank

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert A. DiTomasso .

Additional files

Additional file 1: table s3..

Comparisons overall and by time, location, individual HCCs vs. 3 components. (DOC 41 kb)

Additional file 2: Table S4.

Patient demographics vs. three factors. *refers to higher mean score; ns, not significant. (DOC 44 kb)

Additional file 3: Table S5.

Physician demographics vs. three factors. *refers to higher mean score; ns, not significant. (DOC 40 kb)

Additional file 4: Table S6.

Open-ended questions by healthcare center and year. (+) refers to positive statements, what did you like most? (−) refers to negative statements, what did you like least? (DOC 48 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Thornton, R.D., Nurse, N., Snavely, L. et al. Influences on patient satisfaction in healthcare centers: a semi-quantitative study over 5 years. BMC Health Serv Res 17 , 361 (2017). https://doi.org/10.1186/s12913-017-2307-z

Download citation

Received : 03 April 2017

Accepted : 11 May 2017

Published : 19 May 2017

DOI : https://doi.org/10.1186/s12913-017-2307-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Patient satisfaction
Health care delivery
Community health

BMC Health Services Research

ISSN: 1472-6963

General enquiries: [email protected]

Open access
Published: 18 June 2024

Patient-reported outcomes and experiences of migrants enrolled in a multidisciplinary HIV clinic with rapid, free, and onsite treatment dispensation: the ‘ASAP’ study

Anish K. Arora 1 , 2 , 3 , 4 ,
Serge Vicente 2 , 4 ,
Kim Engler 2 , 4 ,
David Lessard 2 , 4 ,
Edmundo Huerta 2 , 4 ,
Joel Ishak 2 , 4 ,
Nadine Kronfli 2 , 5 ,
Jean-Pierre Routy 5 ,
Joseph Cox 5 , 6 ,
Benoit Lemire 7 ,
Marina Klein 5 ,
Alexandra de Pokomandy 1 , 5 ,
Lina Del Balso 5 ,
Giada Sebastiani 2 , 3 , 5 ,
Isabelle Vedel 1 , 8 ,
Amélie Quesnel-Vallée 6 , 9 ,
Bertrand Lebouché 1 , 2 , 3 , 4 , 5 &

ASAP Migrant Advisory Committee

AIDS Research and Therapy volume 21 , Article number: 40 ( 2024 ) Cite this article

Metrics details

Scholars recommend providing migrants living with HIV (MLWH) with free treatment, rapidly, once linked to care to optimize their HIV-related experiences and health outcomes. Quantitative evaluations of patient-reported measures for MLWH in such models are necessary to explore the viability of these recommendations.

Within a 96-week prospective cohort study at a multidisciplinary HIV clinic, participants received bictegravir/emtricitabine/tenofovir alafenamide (B/F/TAF) for free and rapidly following care linkage. Eight patient-reported measures were administered at weeks 4, 24, and 48: (1) mMOS-SS to measure perceived social support; (2) IA-RSS to measure internalized stigma; (3) K6 to measure psychological distress; (4) PROMIS to measure self-efficacy with treatment taking; (5) G-MISS to measure perceived compliance with clinicians’ treatment plans; (6) HIVTSQ to measure treatment satisfaction; (7) CARE to measure perceived provider empathy; and (8) PRPCC to measure perceived clinician cultural competence. Linear mixed modelling with bootstrapping was conducted to identify significant differences by sociodemographics and time.

Across weeks 4, 24, and 48, results suggest that MLWH enrolled in this study experienced moderate levels of social support; elevated levels of HIV-related stigma; moderate levels of distress; high self-efficacy with daily medication self-management; great compliance with clinicians’ treatment plans; high treatment satisfaction; high perceived empathy; and high perceived cultural competence. Experience of social support (i.e., mMOS-SS scores) differed significantly by birth region. Experience of HIV-related stigma (i.e., IA-RSS scores) differed significantly by birth region, age, and language. Experience of distress (i.e., K6 scores) differed significantly by sexual orientation. Experience of treatment satisfaction (i.e., HIVTSQ scores) differed significantly by birth region and age. No significant differences were identified by time for any measure.

Overall, participants expressed positive experiences around treatment and care, alongside comparably lower perceptions of social support, internalized stigma, and distress, potentially underscoring a need to embed targeted, well-funded, and accessible mental health support within HIV care models.

Introduction

The HIV field has been a champion in progressing global thought and action towards developing models of care that focus on the lived experiences, needs, and preferences of people and populations [ 1 ]. This, in turn, has encouraged the design and implementation of patient-centered health systems for people living with HIV (PLWH) [ 1 , 2 , 3 ] which: advance a holistic understanding of HIV and the multifaceted challenges PLWH present with; reorient the focus of HIV care and research efforts to go beyond simple survival, and instead strive to ensure that PLWH also thrive in their lives; and ultimately, to develop, scale-up, and optimize models of care which allow for sustained and meaningful engagement [ 1 , 4 , 5 ]. Alongside adopting patient-centric approaches, HIV scholars have called for an equity-focused approach to ending the HIV epidemic, whereby efforts are systematically targeted to specific populations with the heaviest burden of HIV [ 6 , 7 ].

People who relocate temporarily or permanently across international borders for any reason (henceforth ‘migrants’), particularly to member countries of the Organization for Economic Co-Operation and Development (OECD), require specific considerations from HIV specialists [ 8 ]. Migrants experience a high burden of HIV and account for a large proportion of new HIV incidence across OECD countries [ 9 , 10 , 11 , 12 ]. For example, in 2020, 44% of those diagnosed with HIV in Europe were migrants, many of which are suggested to have acquired HIV after arrival in the European Union / European Economic Area [ 12 ]. Similarly, in 2020, migrants accounted for 45% of new HIV diagnoses in Canada, of which 54% were diagnosed after their arrival [ 13 ]. Migrants living with HIV (MLWH) encounter numerous intersectional barriers which hinder their access to and engagement with HIV care [ 8 , 14 ]. For instance, MLWH can experience lack of secure and sufficient housing, food, income, legal status, social networks, knowledge around health system navigation, language proficiency, and mental health support [ 8 ]. Additionally, experiences and perceptions of stigma based on one’s HIV and migrant statuses can potentially intersect and amplify the perceived vulnerability of MLWH, further hindering their engagement with HIV care and treatment [ 8 ].

To potentially alleviate challenges faced by MLWH at the clinical level, and thereby improve HIV-related health outcomes, previous work with MLWH suggests the importance of providing migrants with free antiretroviral therapy (ART) dispensed on-site, as well as free-of-charge HIV care (i.e., cost-covered blood tests and clinician visits), as soon as possible after HIV diagnosis [ 8 ]. This combination of factors can enable efficient access to care and treatment, particularly for migrant populations who may have just arrived in their host country, may not have immediate access to public health insurance, and may be unfamiliar with their new local health system [ 15 ]. Furthermore, rapid ART initiation has been shown to reduce loss-to-follow-up between HIV testing and treatment initiation, improve retention in care, and reduce time to HIV viral suppression, without compromising safety [ 15 ]. Additionally, previous research with MLWH suggests the importance of care provision for MLWH through a multidisciplinary team which adopts patient-centric values [ 8 , 15 ]. In the context of HIV, multidisciplinary approaches to care have been associated with numerous clinical and patient-reported advantages relative to standard of care, including higher rates of retention in care, HIV treatment adherence, and improved CD4 counts [ 16 , 17 ], and have been discussed as important by MLWH in meeting their bio-psycho-social needs [ 15 ].

While previous qualitative work indicates that ART, provided rapidly and within a patient-centered multidisciplinary clinic, is well received by MLWH and seems to encourage their initial and sustained engagement with HIV care and treatment [ 15 ], quantitative evidence supporting this approach to care is lacking. More specifically, quantitatively evaluating patient-reported outcomes and experiences at several timepoints throughout the HIV care continuum, including at the early retention phase (< 6 months) and long-term retention phase (> 6 months), has been previously reported as important for studies with vulnerable populations such as MLWH [ 5 ]. Furthermore, assessing variation in patient-reported outcomes and experiences over time and by sociodemographic factors may support the identification of certain profiles of migrants that may experience more challenges and thus require more support within HIV primary care settings. The specific concepts that have been previously suggested as important in exploring for MLWH are: perceived social support, HIV-related internalized stigma, and psychological distress; treatment self-efficacy, compliance, and satisfaction; and provider empathy and cultural competence through patient-reported measures administered over the course of care engagement [ 8 , 15 ]. As such, in this study, we sought to explore patient-reported outcomes and experiences on these concepts among MLWH enrolled in a multidisciplinary program with free, rapid, and onsite ART dispensation.

Study objectives

The objectives of this study are:

To measure participants’ perceived social support, HIV-related internalized stigma, and psychological distress; treatment self-efficacy, compliance, and satisfaction; and provider empathy and cultural competence through patient-reported measures administered over the course of care engagement.

To determine whether differences exist for patient-reported outcomes and experiences by sociodemographic factors and time.

Study design & setting

In January 2020, we initiated a 96-week pilot feasibility study with a prospective cohort design (the ‘ASAP’ Study) at the Chronic Viral Illness Service of the McGill University Health Centre (CVIS/MUHC). The CVIS/MUHC is a public quaternary hospital-based clinic in Montreal, Canada, and serves the largest proportion of MLWH in the city. The CVIS/MUHC offers multidisciplinary HIV care through a team of HIV-specialist physicians, nurses, pharmacists, a social worker, a psychologist, and a psychiatrist. In this study, all participants were initiated on bictegravir/emtricitabine/tenofovir alafenamide (B/F/TAF) rapidly (i.e., within a median of 5 days) after linkage to our clinic [ 18 ]. This manuscript presents an interim analysis of patient-reported quantitative data collected from MLWH enrolled in the ASAP study up to week 48.

Sample size

The ASAP Study’s target population consisted of new treatment-naïve PLWH at the CVIS/MUHC. Notably, the CVIS/MUHC received an average of 30 new ART-naïve MLWH annually between 2016 and 2022. For this interim analysis, 39 migrants had been enrolled in the ASAP Study since January 2020, however, 4 were either lost-to-follow-up or left the study before initiating B/F/TAF and beginning data collection. As no data were available for these migrants, analyses were completed with 35 participants. Note that all 35 participants had been enrolled in the study for at least 24 weeks and 75% had been enrolled for 48 weeks. As a non-probabilistic sampling method was used, no formal sample size calculation based on power considerations and effect sizes was done. However, it is important to note that pilot feasibly studies generally have a sample size of approximately 30 participants on average per intervention arm [ 18 , 19 ]. This small sample of participants does not compromise comparisons between groups of interest. This is because a minimum of 5–10 units per group in longitudinal studies is recommended for group comparisons to assist in avoiding convergence problems which may bias parameter estimates [ 20 , 21 , 22 ].

Data collection

Participants’ sociodemographic characteristics were captured at enrollment and were updated at Week 48. Sociodemographic factors include: birth region, birth year, sex, sexual orientation, living status (i.e., living alone or with others), educational level, occupational status, fluency with French (i.e., the official language of the province), health coverage, and time in Canada before being linked to the CVIS/MUHC. Data was also collected on participants’ usage of SIDEP + , which is a public integrated screening and prevention service for HIV and other sexually transmitted blood-borne infections, for conducting blood test between ASAP study visits. This is because blood tests at the CVIS/MUHC are only covered for those who have access to the provincial medicare system (RAMQ) or a collective insurance plan which covers the cost of HIV care, whereas SIDEP + provides lab tests free of charge for all residents and visitors of Quebec.

Four patient-reported outcome measures (PROMs) and four patient-reported experienced measures (PREMs) were administered at weeks 4, 24, and 48 of the study. PROMs are defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else” [ 23 ]. Alternatively, PREMs provide information on “patients’ perceptions of their experience while receiving care” [ 24 ]. They concern impacts of the process of care, not its outcomes; and they indirectly inform on care quality , not care effectiveness. The eight previously validated patient-reported measures utilized in this study focus on three thematic areas: (1) psychosocial vulnerabilities (i.e., perceived social support, HIV-related internalized stigma, and psychological distress); (2) experience with treatment (i.e., perceived treatment self-efficacy, compliance, and satisfaction); and (3) perceptions of healthcare providers (i.e., perceived clinician cultural competence and empathy). Descriptions for the eight measures, including the way they are scored and details around their validation follow. Note that all questionnaires were administered in French, English, and/or Spanish depending on which language participants preferred. All data were housed in a data management system (i.e., REDCap). Regular quality control checks were carried out to ensure proper data input into and data export from REDCap, the last one between May 23–29, 2024.

Psychosocial vulnerabilities

PROM 1 – Perceived social support was measured with the modified 8-item Medical Outcomes Study Social Support Survey (mMOS-SS) [ 25 , 26 ]. For this PROM, a global score was calculated as the average score of all items, transformed to a 0 to 100 scale [ 26 ]. Higher scores suggest more perceived support. Cronbach’s alpha for the complete scale ranges from 0.88 to 0.93 [ 25 ].

PROM 2 – Perceived HIV-related internalized stigma was measured with the 6-item Internalized AIDS-Related Stigma Scale (IA-RSS) [ 27 ]. A seventh item (“I feel uncomfortable taking my medication in front of someone else”) was added to this PROM to further explore internalized stigma. To simplify administration, the items were dichotomized (1 = Agree, 0 = Disagree). A global score was calculated as the sum of all items (range 0 to 7). Higher scores suggest more internalized stigma. Cronbach’s alpha for the original 6-item scale ranges from 0.73 to 0.76 [ 27 ].

PROM 3 – Psychological distress in the past 30 days was examined with the 6-item Kessler Psychological Distress Scale (K6) [ 28 ]. The 5-point response scale for this PROM ranges from “None of the time” (= 1) to “All of the time” (= 5). Responses are summed to provide a global score, ranging from 6 to 30. Scores of 19 to 30 suggest “a probable serious mental illness” and scores of 6 to 18, “a probable absence of serious mental illness” [ 28 ]. A recent calculation of Cronbach’s alpha is 0.86 [ 29 , 30 ].

Treatment self-efficacy, compliance, and satisfaction

PROM 4 – Treatment self-efficacy (i.e., daily medication self-management) was measured with the PROMIS Self-efficacy for Managing Chronic Conditions – Managing Medications and Treatment – Short Form 4a [ 31 ]. This PROM contains 4-items answered on a 5-point Likert scale from “I am not at all confident” (= 1) to “I am very confident” (= 5). The global score is calculated by summing responses to all items (score range: 4 to 20). Higher scores suggest better self-efficacy. Cronbach’s alpha is between 0.85 and 0.92 [ 31 ].

PREM 1 – Perceived compliance with their clinicians’ treatment plans was measured with a subscale of the Generic Medical Interview Satisfaction Scale (G-MISS) [ 32 ]. The 2 items of the compliance subscale are scored on a 6-point Likert scale from 1 to 6 (i.e., “strongly disagree” to “strongly agree”). A score is obtained for this PREM by calculating the mean of the two items and then transforming the scores into a 0 to 100 scale. Higher sores suggest greater compliance. The subscale has a Cronbach’s alpha of 0.84.

PREM 2 – Treatment satisfaction was measured with the 10-item HIV Treatment Satisfaction Questionnaire (HIVTSQ) – status version [ 33 ]. Items for this PREM are rated from 1 to 7 (1 = least satisfied; 7 = most satisfied), with response options adjusted to the item. The sum of the 10 item scores produces the global scale score (range: 10 to 70). Higher scores suggest greater satisfaction. The measure has a Cronbach’s alpha of 0.91.

Perceptions of healthcare providers

PREM 3 – Perceived provider empathy was measured with the 10-item Consultation and Relational Empathy measure (CARE) [ 34 ]. Items for this PREM are scored on a 5-point rating scale from ‘poor’ (= 1) to ‘excellent’ (= 5). The item ratings are summed to produce the global score (range: 10–50). Higher scores suggest greater perceived empathy. The measure has a Cronbach’s alpha of 0.93.

PREM 4 – Perceived cultural competence of clinicians by patients was measured with the ‘Explaining’ subscale of the Physician Cultural Competency measure (PRPCC) [ 35 , 36 , 37 ]. This PREM includes 8 items which are rated on a 5-point Likert scale (1 = Never to 5 = Always). The global score is computed with the mean score for all 8-item, transformed to 0 to 100. Higher scores suggest greater perceived cultural competence. The complete scale has a Cronbach’s alpha of 0.89.

Data analysis

All quantitative analyses were conducted using R Statistical Software . Means and standard deviations were calculated for each self-reported measure by timepoint and sociodemographic factor. Note that time was treated as a continuous variable in this study. Following descriptive analyses, linear mixed models were fitted to the data [ 38 ]. Linear mixed models are well suited for analyzing longitudinal data with small sample sizes [ 39 ]. Sociodemographic characteristics at enrollment were used for analyses at Weeks 4 and 24, and the updated sociodemographic characteristics were used for analysis at Week 48. The following characteristics were considered to have a fixed effect (i.e., these variables have a constant and consistent influence on the patient-reported measures for all individuals within a particular group): birth region, birth year, sex, sexual orientation, and time in Canada before first visit to the CVIS/MUHC. The other characteristics (i.e., living status, education level, occupational status, fluency with French, health coverage, and SIDEP + usage for blood tests) were considered to have a mixed effect (i.e., these variables may have both a constant and varying influence on the outcome across individuals within groups). To identify the most appropriate analytical model, the Maximum Likelihood Estimation approach was utilized [ 21 , 40 , 41 , 42 ]. The model with the lowest Akaike Information Criterion score for each self-reported measure was chosen and subjected to the Restricted Maximum Likelihood (REML) approach [ 21 , 40 , 41 , 42 ]. To reduce bias introduced by the non-probabilistic sampling method and to enhance generalizability of the results, REML parameters were estimated using a non-parametric bootstrap resampling approach for computing p-values [ 43 , 44 , 45 ]. The bootstrap method is particularly useful when the sample size is insufficient for accurate statistical inference or when selection bias is a concern [ 21 , 43 , 44 , 45 ]. Specifically, we bootstrapped 10,000 samples. Bootstrapped p-values are reported, with a significance level set at < 0.05.

Patient and stakeholder engagement

This study is grounded in patient-oriented research which focuses on: engaging patients and relevant stakeholders as partners, responding to patient-identified priorities, and ultimately improving patient outcomes [ 46 ]. During the ASAP Study, an advisory committee (the ASAP Migrant Advisory Committee), was developed [ 8 , 14 , 15 ]. Members of the ASAP Migrant Advisory Committee contributed to the revision and editing of this manuscript.

This study was conducted in accordance with applicable Health Canada regulations, International Conference on Harmonisation guidelines on current Good Clinical Practice, and the Declaration of Helsinki. It was approved by the Research Ethics Board of the Research Institute of the McGill University Health Centre (reference #: MP-37-2020-4911).

Sociodemographic characteristics

At enrollment, more than half of the participants: came from Africa and/or the Caribbean (n = 20, 57%); were 35 or older (n = 20, 57%); were male (n = 28, 80%); identified as gay, lesbian, or bisexual with respect to their sexual orientation (n = 22, 63%); lived with others (n = 27, 77%); had university-level education (n = 20, 57%); were unemployed (n = 24, 69%); did not speak French (n = 20, 57%); had sufficient health coverage for HIV-related needs through public health insurance (n = 20, 57%); used SIDEP + for at least one blood test (n = 13, 37%); and spent less than 1 year in Canada before being linked to the CVIS/MUHC (n = 20, 57%). These values remained relatively consistent at Week 48. Descriptive statistics by sociodemographic factor at enrolment and at Week 48 are provided in Table 1 .

Social support

The mMOS-SS mean scores (and standard deviations) for the entire sample were 59.4 (26.5), 65.5 (26.5), and 52.6 (30.4) at weeks 4, 24, and 48 respectively (Table 2 ). These scores suggest that on average, throughout the 48 weeks, participants perceived having moderate levels of social support. Significant differences were identified by birth region, where those from Africa and/or the Caribbeans perceived having less social support compared to people from other regions (p = 0.03). No significant differences were identified by the remaining sociodemographic characteristics or time (Table 3 ).

Internalized HIV-related stigma

The IA-RSS mean scores (and standard deviations) for the entire sample were 4.4 (2.0), 3.9 (2.3), and 4.1 (2.3) at weeks 4, 24, and 48 respectively (Table 2 ). These scores suggest that on average, throughout the 48-weeks, participants experienced elevated levels of internalized HIV-related stigma. Significant differences were identified by: birth region, where those from Africa and/or the Caribbeans perceived higher levels of internalized stigma compared to people from other regions (p = 0.002); age, where those less than 35 perceived higher levels of internalized stigma compared to those 35 and older (p = 0.0007); and French fluency, where those not fluent in French perceived higher levels of internalized stigma compared to those fluent in French (p = 0.0033). No significant differences were identified by the remaining sociodemographic characteristics and time (Table 3 ).

Psychological distress

The K6 mean scores (and standard deviations) for the entire sample were 12.5 (5.2), 11.4 (5.1), and 11.5 (5.3) at weeks 4, 24, and 48 respectively (Table 2 ). These scores suggest that on average, throughout the 48 weeks, participants did not experience levels of psychological distress that were indicative of a serious mental illness. Significant differences were identified by sexual orientation, where those who identified as gay or bisexual experienced higher distress compared to those who identified as heterosexual (p = 0.0021). No significant differences were identified by the remaining sociodemographic characteristics or time (Table 3 ).

Treatment-self-efficacy

The PROMIS Self-efficacy mean scores (and standard deviations) for the entire sample were 16.7 (4.2), 17.3 (3.5), and 16.5 (3.7) at weeks 4, 24, and 48 respectively (Table 4 ). These scores suggest that on average, throughout the 48 weeks, participants felt high self-efficacy with respect to daily medication self-management. No significant differences were identified by sociodemographic characteristics or time (Table 3 ).

The G-MISS compliance subscale mean scores (and standard deviations) for the entire sample were 88.4 (23.0), 77.6 (28.4), and 81.6 (23.6) at weeks 4, 24, and 48 respectively (Table 4 ). These scores suggest that on average, throughout the 48 weeks, participants perceived great compliance with their clinicians’ treatment plans. No significant differences were identified by sociodemographic characteristics or time (Table 3 ).

Treatment satisfaction

The HIVTSQ mean scores (and standard deviations) for the entire sample were 62.3 (6.5), 63.9 (6.1), and 61.3 (6.8) at weeks 4, 24, and 48 respectively (Table 4 ). These scores suggest that on average, throughout the 48 weeks, participants felt high satisfaction with their treatment. Significant differences were identified by: birth region, where those from Africa and/or the Caribbeans had lower treatment satisfaction compared to those from other regions (p = 0.0008); and age, where those less than 35 had lower treatment satisfaction compared to those 35 and older (p = 0.0057). No significant differences were identified by the remaining sociodemographic characteristics and time (Table 3 ).

Perceptions around healthcare providers

Provider empathy.

The CARE mean scores (and standard deviations) for the entire sample were 45.1 (6.1), 46.5 (7.0), and 47.5 (3.8) at weeks 4, 24, and 48 respectively (Table 5 ). These scores suggest that on average, throughout the 48 weeks, participants perceived high levels of empathy from their healthcare providers. No significant differences were identified by sociodemographic characteristics or time (Table 3 ).

Provider cultural competence

The PRPCC explaining subscale mean scores (and standard deviations) for the entire sample were 92.0 (10.5), 93.3 (9.9), and 89.1 (14.7) at weeks 4, 24, and 48 respectively (Table 5 ). These scores suggest that on average, throughout the 48 weeks, participants perceived high levels of cultural competence from their clinicians. No significant differences were identified by sociodemographic characteristics or time (Table 3 ).

This study explores the patient-reported outcomes and experiences of MLWH enrolled in a prospective cohort study in Montreal, Canada, where B/F/TAF was being dispensed free-of-charge, onsite, and rapidly after linkage to multidisciplinary HIV care. Specifically, across weeks 4, 24, and 48, self-reported measures were used to assess perceived social support, internalized HIV-related stigma, and psychological distress; treatment compliance, self-efficacy, and satisfaction; and participant perceptions around their healthcare providers’ cultural competence and empathy. To our knowledge, this is the first study that provides quantitative insights on these concepts through self-reported measures among MLWH enrolled in such a model of primary HIV care.

Throughout follow-up, there was a low probability that MLWH experienced a serious mental illness based on their K6 psychological distress scores. However, those who identified as gay or bisexual experienced higher levels of distress. Furthermore, MLWH expressed elevated levels of internalized HIV-related stigma and moderate levels of social support. Notably, those who were from Africa or the Caribbean perceived having less social support. Additionally, those from Africa or the Caribbean, those who were less than 35, and those who were not fluent in French experienced a significantly higher degree of internalized stigma. To some extent, these findings are consistent with those of other studies conducted in Canada and other regions among MLWH, PLWH, and general populations of international migrants [ 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 ]. It is well recognized that migrants often experience higher levels of stigma, mental illness, and challenges with accessing, building, and maintaining social support [ 51 ]. The further burden of living with HIV can amplify these challenges [ 8 , 15 , 50 , 52 , 53 ]. For example, whereas migrants in general struggle with obtaining legal status in their new country, newly-diagnosed MLWH express heightened concern around deportation as a result of stigma, discrimination, and fear from their positive diagnosis [ 8 , 15 ]. Moreover, when additional intersectional burdens are experienced, such as discrimination and stigma due to skin colour or race, or when self-perceived limitations are identified (e.g., lack of ability to speak proficiently in the host nation’s language), levels of internalized stigma among PLWH can increase [ 8 , 54 ]. Also, several studies have previously noted that younger PLWH may experience higher levels of stigma compared to older PLWH [ 50 , 55 , 56 ]. The relationship between age and stigma among MLWH may be attributed to the different life-stages people occupy (e.g., international student versus an established professional), the social networks people have established at different ages, and the coping mechanisms that people have developed and strengthened over time [ 50 ]. While engaged in this cohort study, psychosocial vulnerabilities were not found to decrease significantly over time. This may be linked to the idea that the first year of moving to a new country, learning about one’s HIV diagnosis, and engaging in care and treatment, can be a very challenging time across the emotional, mental, and social levels for individuals [ 15 , 57 , 58 ]. Additionally, perhaps more time (e.g., over 1–2 years) is needed to see a change in these aspects of participants’ lives. Though self-reported measures, like those used in this study, may be helpful in identifying patient perspectives, long time lags are noted to exist for health effects to manifest when dealing with changes to upstream social determinants of health [ 59 ].

Throughout the 48-weeks, participants reported a high degree of treatment self-efficacy, compliance, and satisfaction. Given that some scholars suggest that treatment self-efficacy can be significantly affected by mental health challenges, this finding is unique and important [ 60 ]. In previous qualitative work with MLWH enrolled in the ‘ASAP’ study, MLWH expressed high satisfaction with B/F/TAF, and noted the importance of feeling control over their HIV, as well as a strong sense of responsibility for managing their HIV [ 15 ]. Alongside these individual characteristics, the adoption of person-centered approaches to care may be central to enabling a high degree of daily medication self-management [ 15 ]. Importantly, while levels of treatment self-efficacy, compliance, and satisfaction where high overall in this study, it was also found that those born in Africa and/or the Caribbean and those who were less than 35 years of age had a significantly lower level of satisfaction. The relationship between social factors and treatment satisfaction is complex and not well explored, particularly in the context of HIV among migrant populations. One study describes that racial and ethnic differences in satisfaction may occur based on differences in attitudes and expectations, particularly around patients’ trust with medical care systems [ 61 ]. Another study suggests that older patients may be more satisfied with their healthcare potentially due to generational factors (e.g., those raised during certain periods, such as the early days of the HIV pandemic, may be more experienced with significant hardships, and thus more accepting of inadequacies in healthcare systems) [ 62 ]. Furthermore, findings in this study may be associated with the intersectional challenges that affect these sub-populations’ psychosocial vulnerabilities [ 8 ], but a thorough qualitative exploration is warranted to better understand these phenomena.

Throughout the 48-week period, MLWH perceived high empathy and cultural competence from their clinical team at the CVIS/MUHC. MLWH encompass a diverse group of people, from different ethnicities, cultures, and regions. Despite this, no significant difference was identified by birth region or any other sociodemographic factor with respect to these variables. This finding is interesting given that previous studies have reported poorer satisfaction with healthcare services among migrant populations compared to native-born populations [ 63 , 64 ]. Perhaps this reflects the CVIS/MUHC staff’s experience with working with MLWH. Indeed, these results validate earlier qualitative findings in which MLWH that were receiving care at the CVIS/MUHC discussed their experience of humanizing clinical encounters [ 15 ]. These encounters were characterized by feelings of kindness, acceptance, respect, safety, and trust from and with their clinicians [ 15 ]. Literature suggests that when clinicians adopt such qualities with their patients, they can help promote better rapport-building, higher quality of care, and higher levels of medication self-efficacy [ 15 , 65 , 66 , 67 ]. Additionally, these humanizing qualities are considered essential to develop and sustain people-centered health systems [ 68 ]. However, despite rating their clinical team highly in empathy and cultural competence, MLWH’s psychosocial challenges persisted to some extent across the analytical period. This suggests that humanizing care must be coupled with specific interventions to thoroughly understand and address the complex psychosocial challenges MLWH present with.

Strengths and limitations

A major limitation in this study is the small sample size. As this study was initiated in Jan 2020, a large portion of study recruitment took place during the COVID-19 pandemic. As a result of the limited sample, migrants originating from Africa and the Caribbean were grouped together in the linear mixed modelling analysis. Though previous work in the HIV field has grouped African, Caribbean, and Black populations based on ethnicity and other intersectional challenges experienced by these groups, it is important to acknowledge that Africa and the Caribbean are geographically separate areas of the world, and there can often be large heterogeneity within populations coming from these regions. Another limitation in this study is the use of interim data (i.e., the analysis presented in this study pertains to the halfway point of the 96 week-long ASAP cohort study). However, interim analyses in longitudinal clinical studies, as presented here, are reliable and rational approaches to report findings without comprising validity or integrity [ 69 ]. Such analyses are important for making data and summarized findings available to target audiences in a timely manner, as well as guiding the potential termination or appropriate modifications in sample size or study design [ 69 ]. Notably, by engaging in this process at the mid-point of the ASAP study, we were able to ensure the quality and rigor of our data collection and analysis. Another potential limitation in this study is that the IA-RSS scale was modified by adding one item to further explore internalized stigma. Though we have detailed the item we added in the methods section, and its relevance is warranted; its addition does affect the validity of the IA-RSS scale. Given that the objectives of this study were to measure participants’ self-reported outcomes and experiences, and explore differences by sociodemographic factors and time, the actual utilization of healthcare services (e.g., the number of times patients accessed social worker services) were not examined. Additionally, only a small number of female MLWH agreed to join this cohort study. This is a frequently encountered challenge in HIV clinical research [ 70 ] and has been previously reported by our team [ 15 ]. However, linear mixed modelling analysis is well suited for small samples, and bootstrapping further helps attenuate the small sample size’s effect. Furthermore, the repeated measurement approach (i.e., conducting evaluations at week 4, 24, and 48) and consideration of changes in sociodemographic factors at week 48 allowed for a more rigorous data analysis.

Conclusions

To our knowledge, this is the first study that longitudinally explores the perspectives of MLWH around their perceived social support, internalized HIV-related stigma, and psychological distress; treatment compliance, self-efficacy, and satisfaction; and participant perceptions around their healthcare providers’ cultural competence and empathy. Importantly, findings suggest that most MLWH enrolled in this study expressed high self-efficacy, compliance, and satisfaction with their treatment, and concurrently perceived high cultural competency and empathy from their clinical care providers. Perceived social support, internalized stigma, and distress, however, could be improved among MLWH. In this regard, special attention should be given to people originating from Africa and/or the Caribbean, those less than 35, those not fluent in the native language of their host province, and those identifying as gay or bisexual. These findings potentially underscore the need to embed targeted, well-funded, and accessible mental health support within HIV care models, and that further research is required to better understand how to meet the complex and multifaceted psychosocial needs of MLWH in clinical settings.

Availability of data and materials

Data can be accessed upon reasonable request by contacting the corresponding author.

Joint United Nations Programme on HIV/AIDS. The Greater Involvement of People Living with HIV and/or AIDS. Policy Brief. https://data.unaids.org/pub/briefingnote/2007/jc1299_policy_brief_gipa.pdf . Accessed 31 Jul 2023.

Lazarus JV, Janamnuaysook R, Caswell G. A people-centred health system must be the foundation for person-centred care in the HIV response. J Int AIDS Soc. 2023. https://doi.org/10.1002/jia2.26125 .

Article PubMed PubMed Central Google Scholar

Miles A, Asbridge J. Person-Centered Healthcare-moving from rhetoric to methods, through implementation to outcomes. Eur J Pers Cent Healthc. 2017;5(1):1–9.

Google Scholar

Lazarus JV, Safreed-Harmon K, Barton SE, Costagliola D, Dedes N, del Amo VJ, Gatell JM, Baptista-Leite R, Mendão L, Porter K, Vella S. Beyond viral suppression of HIV–the new quality of life frontier. BMC Med. 2016;14(1):1–5.

Article Google Scholar

Arora AK, Lessard D, Dubé K, Lebouché B. Brief report: it’s time to embed health-related quality of life across all steps of the HIV care cascade. JAIDS J Acquir Immune Defic Syndr. 2022;91(3):251–4.

Article PubMed Google Scholar

Jungwirth, B. Ending the HIV Epidemic: Researchers Call for an Equity-Focused Approach. Coverage of IAS 2021, the 11th International AIDS Society Conference on HIV Science. TheBodyPro For the HIV/AIDS Workforce. https://www.thebodypro.com/article/ending-hiv-epidemic-equity-focused-approach . Accessed 31 July 2023.

Sheikh K, Ranson MK, Gilson L. Explorations on people centredness in health systems. Health Policy Plan. 2014;29:ii1-5.

Arora AK, Ortiz-Paredes D, Engler K, Lessard D, Mate KK, Rodriguez-Cruz A, Kronfli N, Vedel I, Cox J, Quesnel-Vallée A, Antiviral Speed Access Program (ASAP) Migrant Advisory Committee. Barriers and facilitators affecting the HIV care cascade for migrant people living with HIV in organization for economic co-operation and development countries: a systematic mixed studies review. AIDS Patient Care STDs. 2021;35(8):288–307.

Fakoya I, Álvarez-del Arco D, Woode-Owusu M, Monge S, Rivero-Montesdeoca Y, Delpech V, Rice B, Noori T, Pharris A, Amato-Gauci AJ, Del Amo J. A systematic review of post-migration acquisition of HIV among migrants from countries with generalised HIV epidemics living in Europe: implications for effectively managing HIV prevention programmes and policy. BMC Public Health. 2015;15(1):1–4.

Ross J, Cunningham CO, Hanna DB. HIV outcomes among migrants from low-and middle-income countries living in high-income countries: a review of recent evidence. Curr Opin Infect Dis. 2018;31(1):25.

Haddad N, Robert A, Weeks A, Popovic N, Siu W, Archibald C. HIV: HIV in Canada—Surveillance report, 2018. Can Commun Dis Rep. 2019;45(12):304.

Article CAS PubMed PubMed Central Google Scholar

World Health Organization. HIV/AIDS surveillance in Europe: 2021–2020 data.

Public Health Agency of Canada. HIV in Canada, surveillance report to December 31, 2020.

Arora A, Quesnel-Vallee A, Lessard D, Mate K, Rodriguez-Cruz A, Kronfli N, Engler K, Vedel I, Lebouché B. Barriers and facilitators associated with steps of the HIV care cascade for migrants in OECD countries: a systematic mixed studies review protocol. BMJ Open. 2020;10(11):e040646.

Arora AK, Engler K, Lessard D, Kronfli N, Rodriguez-Cruz A, Huerta E, Lemire B, Routy JP, Wittmer R, Cox J, de Pokomandy A. Experiences of migrant people living with HIV in a multidisciplinary HIV care setting with rapid B/F/TAF initiation and cost-covered treatment: the ‘ASAP’ study. J Pers Med. 2022;12(9):1497.

Sherer R, Stieglitz K, Narra J, Jasek J, Green L, Moore B, Shott S, Cohen M. HIV multidisciplinary teams work: support services improve access to and retention in HIV primary care. AIDS care. 2002;14:31–44.

Elgalib A, Al-Sawafi H, Kamble B, Al-Harthy S, Al-Sariri Q. Multidisciplinary care model for HIV improves treatment outcome: a single-centre experience from the Middle East. AIDS Care. 2018;30(9):1114–9.

Arora AK, Vicente S, Engler K, Lessard D, Huerta E, Ishak J, Routy JP, Klein M, Kronfli N, Cox J, Lemire B. Impact of social determinants of health on time to antiretroviral therapy initiation and HIV viral undetectability for migrants enrolled in a multidisciplinary HIV clinic with rapid, free, and onsite B/F/TAF:‘the ASAP study.’ HIV Med. 2024. https://doi.org/10.1111/hiv.13608 .

Beets MW, von Klinggraeff L, Weaver RG, Armstrong B, Burkart S. Small studies, big decisions: the role of pilot/feasibility studies in incremental science and premature scale-up of behavioral interventions. Pilot Feasibility Stud. 2021;7(1):1–9.

Bell BA, Morgan GB, Kromrey JD, Ferron JM. The impact of small cluster size on multilevel models: a Monte Carlo examination of two-level models with binary and continuous predictors. JSM Proc Surv Res Methods Sect. 2010;1(1):4057–67.

McNeish DM, Stapleton LM. The effect of small sample size on two-level model estimates: a review and illustration. Educ Psychol Rev. 2016;28(2):295–314.

Scherbaum CA, Ferreter JM. Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organ Res Methods. 2009;12(2):347–67.

US Department of Health and Human Services FDA Center for Drug Evaluation and Research. 2006 US Department of Health and Human Services FDA Center for Devices and Radiological Health. Guidance for industry: patient‐reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006; 4: 79.

Kingsley C, Patel S. Patient-reported outcome measures and patient-reported experience measures. Bja Educ. 2017;17(4):137–44.

Moser A, Stuck AE, Silliman RA, et al. The eight-item modified medical outcomes study social support survey: psychometric evaluation showed excellent performance. J Clin Epidemiol. 2012;65(10):1107–16.

Rand Corporation. Social support survey instrument scoring instructions. 2021. https://www.rand.org/health-care/surveys_tools/mos/social-support/scoring.html . Accessed 31 July 2023.

Kalichman SC, Simbayi LC, Cloete A, et al. Measuring AIDS stigmas in people living with HIV/AIDS: the Internalized AIDS-Related Stigma Scale. AIDS Care. 2009;21(1):87–93.

Kessler RC, Green JG, Gruber MJ, et al. Screening for serious mental illness in the general population with the K6 screening scale: results from the WHO World Mental Health (WMH) survey initiative. Int J Methods Psychiatr Res. 2010;19(Suppl 1):4–22.

Ferro MA. The psychometric properties of the Kessler Psychological Distress Scale (K6) in an epidemiological sample of Canadian youth. Can J Psychiatry. 2019;64(9):647–57.

Kawakami N, Thi Thu Tran T, Watanabe K, et al. Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam. PLoS ONE. 2020;15(5):e0233119.

Gruber-Baldini AL, Velozo C, Romero S, et al. Validation of the PROMIS(®) measures of self-efficacy for managing chronic conditions. Qual Life Res. 2017;26(7):1915–24.

Maurice-Szamburski A, Michel P, Loundou A, et al. Validation of the generic medical interview satisfaction scale: the G-MISS questionnaire. Health Qual Life Outcomes. 2017;15(1):36–36.

Woodcock A, Bradley C. Validation of the revised 10-item HIV treatment satisfaction questionnaire status version and new change version. Value in Health. 2006;9(5):320–33.

Mercer SW, Maxwell M, Heaney D, et al. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004;21(6):699–705.

Ahmed S, Siad FM, Manalili K, et al. How to measure cultural competence when evaluating patient-centred care: a scoping review. BMJ Open. 2018;8(7):e021525.

Thom DH, Tirado MD. Development and validation of a patient-reported measure of physician cultural competency. Med Care Res Rev. 2006;63(5):636–55.

Glasgow RE, Wagner EH, Schaefer J, et al. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC). Med Care. 2005;43(5):436–44.

Verbeke G, Molenberghs G, Verbeke G. Linear mixed models for longitudinal data. New York: Springer; 1997.

Muth C, Bales KL, Hinde K, Maninger N, Mendoza SP, Ferrer E. Alternative models for small samples in psychological research: applying linear mixed effects models and generalized estimating equations to repeated measures data. Educ Psychol Measur. 2016;76(1):64–87.

Ukyo Y, Noma H, Maruo K, Gosho M. Improved small sample inference methods for a mixed-effects model for repeated measures approach in incomplete longitudinal data analysis. Stats. 2019;2(2):174–88.

Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics. 1997;53:983–97.

Article CAS PubMed Google Scholar

McNeish D. Small sample methods for multilevel modeling: a colloquial elucidation of REML and the Kenward-Roger correction. Multivar Behav Res. 2017;52(5):661–70.

Efron B. Bootstrap methods: another look at the jackknife. Ann Stat. 1979;7(1):1–26.

DiCiccio TJ, Efron B. Bootstrap confidence intervals. Statist Sci. 1996;11(3):189–228.

Adèr HJ, Mellenbergh GJ, Hand DJ. Advising on research methods: a consultant’s companion. Amsterdam: Johannes van Kessel Publishing; 2008.

Canadian Institutes of Health Research. Strategy for Patient-Oriented Research - Patient Engagement Framework. 2019. https://cihr-irsc.gc.ca/e/48413.html . Accessed 31 Jul 2023.

Schweitzer R, Melville F, Steel Z, Lacherez P. Trauma, post-migration living difficulties, and social support as predictors of psychological adjustment in resettled Sudanese refugees. Aust N Z J Psychiatry. 2006;40(2):179–87.

Stewart M, Anderson J, Beiser M, Mwakarimba E, Neufeld A, Simich L, Spitzer D. Multicultural meanings of social support among immigrants and refugees. Int Migr. 2008;46(3):123–59.

Dela Cruz AM, Maposa S, Patten S, Abdulmalik I, Magagula P, Mapfumo S, Abate TD, Carter A, Spies P, Harrowing J, Hall M. “I die silently inside”. Qualitative findings from a study of people living with HIV who migrate to and settle in Canada. J Migr Health. 2022;5:100088.

Emlet CA, Brennan DJ, Brennenstuhl S, Rueda S, Hart TA, Rourke SB. The impact of HIV-related stigma on older and younger adults living with HIV disease: does age matter? AIDS Care. 2015;27(4):520–8.

Blackmore R, Boyle JA, Fazel M, Ranasinha S, Gray KM, Fitzgerald G, Misso M, Gibson-Helm M. The prevalence of mental illness in refugees and asylum seekers: a systematic review and meta-analysis. PLoS Med. 2020;17(9):e1003337.

Wong JP, Li AT, Poon MK, Fung KP. an exploratory study on the mental health of immigrants, refugees and non-status people living with HIV in Toronto. Int J Migr Health Soc Care. 2013;9(3):122–34.

Stutterheim SE, Bos AE, Shiripinda I, de Bruin M, Pryor JB, Schaalma HP. HIV-related stigma in African and Afro-Caribbean communities in the Netherlands: manifestations, consequences and coping. Psychol Health. 2012;27(4):395–411.

Doyal L. Challenges in researching life with HIV/AIDS: an intersectional analysis of black African migrants in London. Cult Health Sex. 2009;11(2):173–88.

Earnshaw VA, Bogart LM, Dovidio JF, Williams DR. Stigma and racial/ethnic HIV disparities: moving toward resilience.

Pantelic M, Casale M, Cluver L, Toska E, Moshabela M. Multiple forms of discrimination and internalized stigma compromise retention in HIV care among adolescents: findings from a South African cohort. J Int AIDS Soc. 2020;23(5):e25488.

LeMaster JW, Broadbridge CL, Lumley MA, Arnetz JE, Arfken C, Fetters MD, Jamil H, Pole N, Arnetz BB. Acculturation and post-migration psychological symptoms among Iraqi refugees: a path analysis. Am J Orthopsychiatry. 2018;88(1):38.

Mohamad Fisal ZA, Minhat HS, Mohd Zulkefli NA, Ahmad N. Biopsychosocial approach to understanding determinants of depression among men who have sex with men living with HIV: a systematic review. PLoS ONE. 2022;17(3):e0264636.

Braveman P, Gottlieb L. The social determinants of health: it’s time to consider the causes of the causes. Public Health Rep. 2014;129:19–31.

Crockett KB, Entler KJ, Brodie E, Kempf MC, Konkle-Parker D, Wilson TE, Wingood G, Neilands TB, Johnson MO, Weiser SD, Turan JM. Linking depressive symptoms to viral non-suppression among women with HIV through adherence self-efficacy and ART adherence. J Acquir Immune Defic Syndr (1999). 2020;83(4):340.

Barr DA. Race/ethnicity and patient satisfaction: using the appropriate method to test for perceived differences in care. J Gen Intern Med. 2004;19(9):937–43.

Jaipaul CK, Rosenthal GE. Are older patients more satisfied with hospital care than younger patients? J Gen Intern Med. 2003;18(1):23–30. https://doi.org/10.1046/j.1525-1497.2003.20114.x .

Mastaki JK. Migrant patients’ satisfaction with health care services: a comprehensive review. Italian J Public Health. 2010;7(1):69.

Kieseppä V, García Velázquez R, Vehko T, Kuusio H. Satisfaction with access to health services among foreign-born population in Finland: a survey-based study. BMC Health Serv Res. 2022;22(1):1–1.

Flickinger TE, Saha S, Roter D, Korthuis PT, Sharp V, Cohn J, Moore RD, Ingersoll KS, Beach MC. Respecting patients is associated with more patient-centered communication behaviors in clinical encounters. Patient Educ Couns. 2016;99(2):250–5.

Flickinger TE, Saha S, Roter D, Korthuis PT, Sharp V, Cohn J, Eggly S, Moore RD, Beach MC. Clinician empathy is associated with differences in patient–clinician communication behaviors and higher medication self-efficacy in HIV care. Patient Educ Couns. 2016;99(2):220–6.

Dapaah JM. Attitudes and behaviours of health workers and the use of HIV/AIDS health care services. Nutr Res Pract. 2016;28:2016.

International Alliance of Patients’ Organizations (IAPO). Patient‐Centred Healthcare Indicators Review.

Kumar A, Chakraborty BS. Interim analysis: a rational approach of decision making in clinical trial. J Adv Pharm Technol Res. 2016;7(4):118.

Mendez KJ, Cudjoe J, Strohmayer S, Han HR. Recruitment and retention of women living with HIV for clinical research: a review. AIDS Behav. 2021;25(10):3267–78.

Download references

Acknowledgements

The authors would like to thank the numerous patients, staff, community organizations, and clinicians who contributed to this research study.

This study is funded in part by the Gilead Investigator Sponsored Research Program (Grant # IN-US-380-4670). This grant was awarded to Dr. Bertrand Lebouché, who holds a Canadian Institutes of Health Research (CIHR), Strategy for Patient-Oriented Research (SPOR) Mentorship Chair in Innovative Clinical Trials for HIV Care. He is supported by two career awards: a Senior Salary Award from Fonds de la Recherche du Québec–Santé (FRQ-S) (#311200) and the LE 250 from Quebec’s Ministry of Health for researchers in Family Medicine. Anish K. Arora is supported by a Vanier Canada Graduate Scholarship from the Canadian Institutes of Health Research.

Author information

Authors and affiliations.

Department of Family Medicine, Faculty of Medicine and Health Sciences, McGill University, Montréal, Québec, Canada

Anish K. Arora, Alexandra de Pokomandy, Isabelle Vedel & Bertrand Lebouché

Centre for Outcomes Research & Evaluation, Research Institute of the McGill University Health Centre, 5252 de Maisonneuve Blvd - Office 3C.25, Montreal, QC, H4A 3S5, Canada

Anish K. Arora, Serge Vicente, Kim Engler, David Lessard, Edmundo Huerta, Joel Ishak, Nadine Kronfli, Giada Sebastiani & Bertrand Lebouché

Infectious Diseases and Immunity in Global Health Program, Research Institute of the McGill University Health Centre, Montréal, QC, Canada

Anish K. Arora, Giada Sebastiani & Bertrand Lebouché

Canadian Institutes of Health Research Strategy for Patient-Oriented Research (CIHR/SPOR) Mentorship Chair in Innovative Clinical Trials in HIV Care, Montréal, Canada

Anish K. Arora, Serge Vicente, Kim Engler, David Lessard, Edmundo Huerta, Joel Ishak & Bertrand Lebouché

Department of Medicine, Chronic Viral Illness Service, Division of Infectious Diseases, Department of Medicine, McGill University Health Centre, Montréal, QC, Canada

Nadine Kronfli, Jean-Pierre Routy, Joseph Cox, Marina Klein, Alexandra de Pokomandy, Lina Del Balso, Giada Sebastiani & Bertrand Lebouché

Department of Epidemiology, Biostatistics and Occupational Health, Faculty of Medicine & Health Sciences, McGill University, Montréal, QC, Canada

Joseph Cox & Amélie Quesnel-Vallée

Pharmacy Department, McGill University Health Centre, Montréal, QC, Canada

Benoit Lemire

Lady Davis Institute, Jewish General Hospital, Montréal, QC, Canada

Isabelle Vedel

Department of Sociology, Faculty of Arts, McGill University, Montréal, QC, Canada

Amélie Quesnel-Vallée

You can also search for this author in PubMed Google Scholar

Contributions

Manuscript conceptualization: Anish K. Arora. Data collection: David Lessard, Edmundo Huerta, Joel Ishak. Data analysis: Anish K. Arora, Serge Vicente. Drafting the manuscript: Anish K. Arora. Manuscript editing: Serge Vicente, Kim Engler, David Lessard, Edmundo Huerta, Joel Ishak, Nadine Kronfli, Jean-Pierre Routy, Joseph Cox, Benoit Lemire, Lina Del Balso, Marina Klein, Alexandra de Pokomandy, Giada Sebastiani, Isabelle Vedel, Amélie Quesnel-Vallée, ASAP Migrant Advisory Committee, Bertrand Lebouché. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Bertrand Lebouché .

Ethics declarations

Competing interests.

Dr. Bertrand Lebouché has received consultancy fees and/or honoraria and research funds from Gilead, ViiV Healthcare, and Merck. The organizations and the above funders had no role in the design of the study; in the collection, analyses, or interpretation of data; or in the decision to publish the results.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Arora, A.K., Vicente, S., Engler, K. et al. Patient-reported outcomes and experiences of migrants enrolled in a multidisciplinary HIV clinic with rapid, free, and onsite treatment dispensation: the ‘ASAP’ study. AIDS Res Ther 21 , 40 (2024). https://doi.org/10.1186/s12981-024-00632-5

Download citation

Received : 26 November 2023

Accepted : 09 June 2024

Published : 18 June 2024

DOI : https://doi.org/10.1186/s12981-024-00632-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Patient-reported outcomes
Care experience
Multidisciplinary

AIDS Research and Therapy

ISSN: 1742-6405

Submission enquiries: [email protected]

June 10, 2024

IMD-mediated innate immune priming increases Drosophila survival and reduces pathogen transmission

Image credit: ppat.1012308

Calling all experts!

Plos pathogens welcomes talented individuals to join our editorial board. .

research article

Herpes simplex virus spreads rapidly in human foreskin, partly driven by chemokine-induced redistribution of Nectin-1 on keratinocytes

Insights into the initial events that lead to HSV1 infection and spread within human genital mucosa and definition of a time window of opportunity to target the virus before it enters epidermal nerves and becomes latent.

Image credit: ppat.1012267

Herpes simplex virus spreads rapidly in human foreskin, partly driven by chemokine-induced redistribution of Nectin-1 on keratinocytes

Recently Published Articles

2+ -dependent association of cysteine-rich protein with virion orchestrates morphogenesis of rod-shaped viruses">Zn 2+ -dependent association of cysteine-rich protein with virion orchestrates morphogenesis of rod-shaped viruses
Mycobacterium tuberculosis -infected Diversity Outbred mice">Systems genetics uncover new loci containing functional gene candidates in Mycobacterium tuberculosis -infected Diversity Outbred mice

Arboviruses antagonize insect Toll antiviral immune signaling to facilitate the coexistence of viruses with their vectors

Current Issue May 2024

The anti-apoptotic function of HSV-1 LAT in neuronal cell cultures but not its function during reactivation correlates with expression of two small non-coding RNAs, sncRNA1&2

LAT resides within the two sncRNAs, which works independently of its latency-reactivation function and it has suppressive effect on LAT expression in vivo and in vitro .

Image credit: ppat.1012307

The anti-apoptotic function of HSV-1 LAT in neuronal cell cultures but not its function during reactivation correlates with expression of two small non-coding RNAs, sncRNA1&2

Novel strategies to evade and even exploit insect Toll antiviral immune response to facilitate persistent viral transmission.

Image credit: ppat.1012318

Arboviruses antagonize insect Toll antiviral immune signaling to facilitate the coexistence of viruses with their vectors

Quantifying the fitness effects of resistance alleles with and without anthelmintic selection pressure using Caenorhabditis elegans

Multiple traits should be considered to understand resistance comprehensively and that the determination of whether a gene plays a role in anthelmintic resistance depends on the trait measured.

Image credit: ppat.1012245

Infectious hepatitis E virus is associated with the mature sperm head

HEV isolated from sperm remained infectious suggesting a potential transmission route via sexual partners.

Image credit: ppat.1012240

Protecting the endothelial glycocalyx in COVID-19

Hyperimmune activation or persistent presence of viral antigens are associated with EG damage and may potentially bring about continuous endothelial dysfunction even after clinical recovery.

Image credit: ppat.1012203

Research Article

The read-through transcription-mediated autoactivation circuit for virulence regulator expression drives robust type III secretion system 2 expression in Vibrio parahaemolyticus

Autoregulatory loop is essential for sustained vtrB activation ...

Image credit: ppat.1012094

Characterization and genomic analysis of the Lyme disease spirochete bacteriophage ϕBB-1

The ubiquitous ϕBB-1 phage and its implications in the generalized transduction of diverse genes and the maintenance of genetic diversity.

Image credit: ppat.1012122

Broad-spectrum Delta-BA.2 tandem-fused heterodimer mRNA vaccine delivered by lipopolyplex

This vaccine has demonstrated its ability to induce immunogenicity in mice and offers protection to both mice and monkeys when exposed to live SARS-CoV-2 challenges.

Image credit: ppat.1012116

Pathobiology and dysbiosis of the respiratory and intestinal microbiota in 14 months old Golden Syrian hamsters infected with SARS-CoV-2

Brittany Seibert, C. Joaquín Cáceres, and colleagues

Bile acids promote the caveolae-associated entry of swine acute diarrhea syndrome coronavirus in porcine intestinal enteroids

Qi-Yue Yang, Yong-Le Yang, and colleagues

Multi-omics analysis of the cervical epithelial integrity of women using depot medroxyprogesterone acetate

Frideborg Bradley and colleagues

Modulation of sensory perception by hydrogen peroxide enables Caenorhabditis elegans to find a niche that provides both food and protection from hydrogen peroxide

Jodie A. Schiffer & Stephanie V. Stumbur and colleagues

Evidence of MHC class I and II influencing viral and helminth infection via the microbiome in a non-human primate

B. Karina Montero and colleagues

Advances in understanding red blood cell modifications by Babesia

Hakimi et al.

After the virus has cleared—Can preclinical models be employed for Long COVID research?

Jansen et al.

Viral-mediated activation and inhibition of programmed cell death

Verburg et al.

Molecular architecture of bacterial type IV secretion systems

Sheedlo et al.

Bidirectional cytokine-microRNA control: A novel immunoregulatory framework in leishmaniasis

Jafarzadeh et al.

Publish with PLOS

Submission Instructions
Submit Your Manuscript

Connect with Us

PLOS Pathogens on Twitter
PLOS on Facebook

Get new content from PLOS Pathogens in your inbox

Thank you you have successfully subscribed to the plos pathogens newsletter., sorry, an error occurred while sending your subscription. please try again later..

Quantitative strategies, Wall Street-caliber research, and insightful market analysis since 1998.

Inclusive Alliance Wins National Grant to Enhance its Community Care Hub Infrastructure

Funding to Strengthen Community Care and Enhance Health Equity Across Central New York

(PRUnderground) June 17th, 2024

The Inclusive Alliance, an Independent Practice Association of health and human service nonprofit agencies serving the counties of Cayuga, Cortland, Madison, Onondaga, and Oswego and beyond, was selected as one of 20 grantees for USAging’s Center of Excellence to Align Health and Social Care (COE). With the funding received through the COE, we will be able to better support the infrastructure of our growing network of community-based organizations. The COE is supported by the U.S. Administration for Community Living (ACL) and will support community care hubs in their work to assess and address individuals’ health-related social needs and promote whole-person care.

The intent of these awards is to provide infrastructure funding to the selected CCHs like Inclusive Alliance in support of their contracting efforts with health care organizations for coordinated access to and delivery of social care programs and services (including support for hospital-to-home care transition programs) that assess and address health-related social needs and improve health equity and inclusion for individuals, families, and/or caregivers. These efforts will also enhance New York’s No Wrong Door access systems and coordinate with local and/or state public health departments.

“This funding represents a significant step forward in our mission to provide better care for our community across Central New York,” said Lauren Wetterhahn, Executive Director of Inclusive Alliance. “With this support, we will be able to strengthen our network and ensure that individuals and families have access to the inclusive, comprehensive services they need. Together, we are building a future where everyone in our community can get access to the services they need to thrive.”

“USAging is thrilled to award grants to these 20 outstanding Community Care Hubs,” said USAging CEO Sandy Markwood. “CCHs provide critical administrative and operational supports for networks of Area Agencies on Aging, Centers for Independent Living and other community-based organizations providing services addressing the health-related social needs of individuals with complex care needs and their caregivers. This funding will bolster the vital work CCHs do in their communities every day to align health and social care to meet people’s needs.” Markwood continued.

“Health care providers and plans are increasingly prioritizing screening for health-related social needs such as food insecurity, housing instability, unaffordable utilities and lack of transportation, and the aging and disability networks are critical partners in connecting people to the services and programs needed to address them,” said Alison Barkoff, who leads ACL. “It has been truly amazing to see the speed at which this work has advanced, thanks in large part to our network’s creation of community care hubs, which are establishing partnerships and contracts between community-based organizations and health care organizations across the nation. ACL looks forward to ongoing collaboration with the Center of Excellence to continue growing community care hubs across the entire country.”

About Inclusive Alliance IPA Inc

Inclusive Alliance is an Independent Practice Association of community-based organizations of varying sizes and scopes of services across Cayuga, Cortland, Madison, Onondaga and Oswego counties and beyond. The mission of the Inclusive Alliance is to advance the growth and quality of cost effective and inclusive services for children and adults through innovation, collaboration and coordination. The Inclusive Alliance’s purpose is to prepare members for managed care and value-based payment. Learn more at inclusivealliance.org.

Press Contact

Original Press Release .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Int J Environ Res Public Health
PMC10341511

The Association of Moral Injury and Healthcare Clinicians’ Wellbeing: A Systematic Review

Associated data.

Data sharing not applicable.

Research focused on elucidating moral injury amongst healthcare workers (HCWs) is essential due to the deep connection with morality and individuals’ overall wellbeing. Examining moral injury provides an avenue through which researchers can connect individual experiences with systemic level causes (i.e., structural power imbalances between clinicians and health systems) to better study workplace wellbeing. The omnipresence of the COVID-19 pandemic has amplified the need to study moral injury. This paper describes a systematic review conducted using PRISMA-P guidelines to answer the question, “what is the association between moral injury and professional wellbeing and mental health amongst healthcare workers.” Twelve databases were searched to identify specified studies. This study’s criteria included: (1) articles published through December 2022; (2) qualitative and quantitative empirical studies; (3) articles written in English; (4) articles including moral injury; and (5) articles including at minimum one other measure of professional or personal wellbeing. The initial search produced 248 articles, and 18 articles were ultimately included in the final review. To confirm that no articles were left out of this study, the first author of each included article was contacted to inquire about any additional works that met the inclusion criteria of this study. The elements of the 18 included articles described in this review are discussed. The results indicate that moral injury is associated with both professional wellbeing factors and mental health outcomes. Further theoretical development, including (professional- and identity-based) exploratory research on moral injury, and evidenced-based interventions for moral injury are needed.

1. Introduction

In the United States, there are 22 million people working within the healthcare system [ 1 ]. In a recent report, “Clinicians of the Future”, only 57% of healthcare workers (HCWs) believed that they have a good work–life balance [ 2 ]. One reason why achieving a good work–life balance in healthcare is challenging is due to the high strain the healthcare system in the United States places on HCWs. There have been many indications of this high strain on HCWs such as experiences of burnout, adverse mental health, moral injury, and high turnover [ 3 , 4 , 5 ]. HCWs experience burnout at a higher rate than other professions [ 4 ], and they also experience higher rates of depression, anxiety, post-traumatic stress disorder (PTSD), or suicide than the general population [ 6 ]. About 47% of U.S.-based clinicians out of nearly 3000 people surveyed in a recent study stated that they intended to leave their job within the next two to three years [ 2 ]. The stress on healthcare workers exacerbates fractures in the services received by patients. For example, when a clinician experiences adverse job-related wellbeing, they are more likely to make a mistake in their work with patients [ 4 ]; whether it is a surgeon making a mistake in the operating room or a social worker making a mistake in creating a safe discharge plan for a patient. Declining wellbeing in healthcare workers is risky and harmful to the quality of care provided in hospitals and to patient related outcomes during and post-hospitalization [ 7 ].

It is well known that HCWs are not doing well and are in search of further support in their roles [ 8 , 9 ]. The COVID-19 global pandemic further strained U.S. healthcare workers by adding to the pressure of their roles through expectations of heroism and sacrifice, demonstrated by amplified rates of patient abuse towards HCWs [ 10 ]. Healthcare work in the pandemic amplified the focus on morality via the question of what “doing the right thing” means. Moral injury was not often discussed in pre-pandemic healthcare discourse; yet, there became an urgent need to better understand the moral and ethical impacts on healthcare workers.

Healthcare workers and policy are present in all systems. Well before COVID-19, schools had nurses, colleges had health and counseling centers, grocery stores had pharmacies, and all of us, when unwell, craved an understanding of how to heal or feel better. Healthcare systems cease to exist without healthcare workers; thus, it is imperative to identify the grave risk of losing healthcare access if our healthcare workers are not being supported as individuals, but rather are treated inhumanely by both the systems and the patients. When addressing human needs regarding wellbeing in high stakes situations such as healthcare, the inclusion of morality is important. A holistic understanding HCWs’ wellbeing can be achieved through studying how moral injury, or harm to individuals’ sense of morality, is associated with wellbeing outcomes amongst the healthcare workforce.

1.1. Moral Injury as a Measure of Wellbeing

Moral injury is operationalized in several different ways across the literature. There are two primary definitions of moral injury that are used in the field of ethics and morality. First, Shay’s work was the original definition of moral injury rooted in military populations, which states, “(a) a betrayal of ‘what’s right’; (b) either by a person in legitimate authority or by oneself; (c) in a high stake’s situation” (p. 182) [ 11 ]. Shay’s definition is strong due to its explicitness and is simultaneously critiqued for being too rigid. Litz and Kerig’s definition, which is more flexible and applicable to a variety of settings and events, describes moral injury as, “transgressive harms and the outcomes of those experiences’” [ 12 ] (p. 341). For this review, a joint definition is used; moral injury is defined as the moral transgression (or boundary breaking) by oneself or someone in a position of power in high stakes situations and the outcomes of those experiences [ 11 , 12 ]. While a range of moral outcomes can be experienced [ 13 ] from a potentially morally injurious event (PMIE), a situation in which ethical dilemmas could lead to a moral transgression [ 12 ], moral injury is a nuanced experience in which a moral transgression actually occurs. Thus, the experience of moral injury is a unique risk factor to additional adverse wellbeing outcomes. The continuum below identifies and differentiates between the various moral reactions one may have while working in healthcare, ranging from moral frustration to moral distress to moral injury (see Figure 1 below). While there are various moral reactions a HCW can experience, this review specifically focuses on the experience of moral injury; thus, separation of terms is necessary. Moral distress, a construct developed by the nursing field, speaks to the experience of internal or external constraints challenging HCW’s judgements on “what’s right” [ 13 , 14 ]. Moral distress can be a prolonged experience, one which does not always result in completing a moral transgression [ 14 , 15 ]. Moral frustration speaks to the emotional reaction of facing a PMIE [ 14 ], and it is the experiences of emotions related to moral challenges that may or may not be directly related to oneself. Figure 1 (below) locates these phenomena across a continuum.

An external file that holds a picture, illustration, etc.
Object name is ijerph-20-06300-g001.jpg

Continuum of Moral Experiences.

A few studies have highlighted the high prevalence of moral injury amongst healthcare workers prior to and during the COVID-19 pandemic [ 16 , 17 , 18 ]. During COVID-19, many HCWs experienced moral injury due to shortages of triage, PPE, medication, and supplies [ 18 ], highlighting the essential role of adequate resources in healthcare worker wellbeing. In another study, the concept of moral injury resonated with medical students who had observed situations contrary to their own values as future medical professionals and felt that they could not live up to the standards that were required of them [ 19 ]. Experiencing moral injury can result in feelings of guilt, shame, and internal confusion with oneself [ 11 , 12 ]. In general, experiencing feelings of guilt and shame is related to overall adverse mental health and wellbeing [ 20 , 21 ]. A further synthesis of how moral injury is then related to professional and personal wellbeing is needed.

1.2. Wellbeing Indicators for Healthcare Workers

Workforce wellbeing has been measured using numerous indices and labels contextualizing the high stakes roles of working in healthcare. For this systematic review, HCWs include all patient-facing professions, such as physicians, nurses, social workers, care aids, psychiatrists, psychologists, pharmacists, and physical, occupational, and speech therapists. Generally, the workforce wellbeing literature is divided into two categories: personal wellbeing and professional wellbeing. In this review, professional wellbeing is operationalized as work-related adverse experiences such as burnout, imposter syndrome, compassion fatigue, and turnover. Meanwhile, personal wellbeing refers to the individualized outcomes that impact how one is doing broadly, such as mental health (depression, anxiety, and PTSD) and stress.

Many studies of HCWs professional wellbeing describe high levels of burnout, compassion fatigue, and secondary trauma amongst healthcare clinicians (referring predominantly to doctors and nurses) [ 22 ]. Healthcare environments are fast-paced, stressful, and high stakes spaces, and the nature of healthcare has led to many adverse professional wellbeing consequences on HCWs. HCWs’ roles are usually high-pressure and under-resourced, often leading to these negative professional wellbeing outcomes [ 23 , 24 , 25 , 26 , 27 ]. Prior to the COVID-19 pandemic, HCWs were already experiencing high rates of negative professional wellbeing [ 27 ], and the global pandemic has only exacerbated these experiences and consequences [ 28 , 29 , 30 ]. Specifically, across several cross-sectional studies HCWs experienced burnout and exhaustion at the rate of 76% and compassion fatigue at the rate of 52% ( n = 1119) [ 31 ].

Further, HCW wellbeing research has illustrated the impact of healthcare work on individuals’ personal wellbeing, primarily referring to mental health. HCWs experience higher levels of mental health diagnoses and related symptoms [ 32 ]. For example, physicians are twice as likely than the general population to die by suicide [ 33 ]. Poor wellbeing and high levels of burnout have been associated with poor patient safety outcomes such as medical errors [ 34 ]. Moreover, HCWs experience high levels of depression, anxiety, and PTSD related to their jobs [ 23 , 28 , 30 ]. According to Mental Health America’s 2020 survey, 39% of healthcare workers did not feel adequately emotionally supported—with nurses even less likely to feel supported (45%) [ 31 ]. Environments with high exposure to stress and trauma, like healthcare, put HCWs at higher risk of experiencing negative personal wellbeing outcomes. Additionally, personal wellbeing includes spirituality and religiosity. How individuals make meaning of the world can often be derived from spirituality. Moral injury is a concept that is rooted in spirituality and religiosity, and hence the measure of religiosity/spirituality is included in research on moral injury [ 11 ].

1.3. Positionality Statement

Prior to reading the methods and findings from this review, it is essential for the authors and coders of this work to identify ourselves in relation to the work. The lead author and primary coder is an educated Indian American, cis-gender woman, first-generation college student, and second-generation immigrant who comes from a family that has struggled to attain financial security. Her positionality is essential to name in the present work, as she has experienced moral injury as a healthcare clinical social worker. The second author and secondary coder on this review is a white, queer-identifying, first generation college graduate. She is the child of actors, an older sibling, and comes from a family history of low-economic status, addiction, and untreated mental illness. The third author on this paper is a white, queer, cis-gender woman with experience as a patient with PTSD in the US. The last author is a white, cis-gender woman who has former experience as a HCW and family members with serious and persistent mental illness and addiction. Together, this research team has experiences with mental health care from the perspectives of patients, family members, and HCWs; these are all essential to name in this study that discusses mental health as a measure of wellbeing.

1.4. Purpose

This systematic review was conducted to rigorously identify and critically analyze the literature on moral injury and wellbeing (personal and professional) in healthcare settings. This review will contribute to future research by providing foundational knowledge on moral injury and its associated outcomes. The primary guiding research question for this review is:

What is the relationship between moral injury and wellbeing (personal and professional) amongst HCWs?

To best answer the research question, this review will include a summary of concepts, theories, methodologies, and results/findings found in prior studies of the association of moral injury and wellbeing.

2. Materials and Methods

2.1. prisma-p protocol overview.

This systematic review is registered with the Open Science Framework ( https://osf.io/q96bp/ accessed on 27 June 2021) and follows the PRISMA-P Systematic Review Protocol [ 35 , 36 ]. Guidelines from Boland and scholars and the Centre for Reviews and Dissemination were followed [ 37 , 38 ]. A mixed methods approach was used for this review including both qualitative and quantitative articles. Two reviewers (PT and AN) followed the same systematic search process to collect data based on the pre-established PRISMA-P protocol. A university-based research librarian supported the development of this protocol in the areas of information sources, inclusion/exclusion criteria, and search strategy.

2.2. Information Sources

The information sources for this review included healthcare-related and spirituality databases that included both peer-reviewed and gray literature. The following databases were searched: Academic Search Complete, ATLA, Dissertations & Theses Full Text, Google Scholar, Open Gray, Philosopher’s Index, PubMed, Religion Database, Social Services Abstract, SocIndex, and Web of Science. To frame the search strategy for this review, the SPICE framework was used to describe the parameters of the study. SPICE represents: the setting (S), which is healthcare settings; the population (P), which includes all HCWs; the interest (I), which is moral injury; comparison (C), though this review did not use a comparison group; and last, evaluation (E), which represents the association examined between wellbeing and the interest of moral injury [ 39 ].

2.3. Inclusion and Exclusion Criteria

The articles included in this review were selected if they met specific parameters focusing on the experience of moral injury in healthcare amongst healthcare workers. The review included both open access and fee-based articles accessible to the University of Denver Library. Dissertations and gray literature were included in the search because of the relatively recent emergence of this area of study. Regarding event history, studies before and during the COVID-19 pandemic were both included in this study. Articles were included if they went through a peer-review process and were empirically based. Differentially, studies that were related to moral injury were included in the study, and studies related to moral distress were excluded from the study. Commentary, conceptual, and review articles were excluded from this review. This study excluded articles that were not in English.

2.4. Search Strategy

The search was conducted using Boolean/phrase operators. The Boolean operators used were: (“moral injur*”) AND (wellbeing or well being or well-being or burnout or burned out or burnt out or compassion fatigue or retention, turnover, mental health, depress*, anxiet*, trauma, or stress) AND (healthcare professional or healthcare clinician or clinician or doctor or physician or nurse or nurse practitioner or physician assistant or social work* or clinical social work* or psychologist or psychiatrist) NOT (military, “active duty”, veteran, army, navy, “air force”).

Once the search protocol was implemented across all sources, the articles were imported into a reference management system, Zotero. All identified titles and abstracts were screened for the inclusion of moral injury and wellbeing outcomes within samples of HCWs. A title and abstract screening form was created to ensure that all articles were consistently screened using the same criteria. Once the articles were screened, conceptual articles, reviews, and letters to the editor were removed from the selected articles.

Prior to extracting data from the articles, PS emailed all the first authors of the included articles. This email requested their consultation, informed them that their article is going to be included in this systematic review, and inquired about any ongoing research about moral injury and wellbeing in healthcare.

2.5. Data Extraction

A data extraction form was created based on Cochrane’s data extraction template [ 40 ] and was used to extract data from the empirical studies included in this review. The same two reviewers, or henceforth “coders”, who systematically searched for the articles also extracted the data from the articles included in this study [ 40 ]. Using a data extraction tool, both coders reviewed and annotated each article in depth. Each coder was responsible for responding to each question in the data extraction tool for each independent article included in this review. After data extraction was completed individually by each coder, both coders met over a series of meetings to compare their data extraction responses. All discrepancies in the data extraction were reinvestigated by the two coders together until an agreement was reached.

2.6. Data Synthesis and Analysis

After the data were accurately extracted from both qualitative and quantitative papers, a standard review table was created to visually synthesize the included articles in this review. The table includes the following information: authors, publication year, location, aims, theoretical framing, sample (size, description, method), study design and methodology, outcome variables, study findings/results, and a conclusion statement about moral injury and wellbeing.

The analysis of this review included a synthesis of both qualitative and quantitative studies. The synthesis includes descriptive study data as well as conceptual and methodological summaries. The review table provides an accessible summary of the current state of research on moral injury and wellbeing, and the research is more deeply critiqued in the following narrative.

Quality Appraisal and Risk-for-Bias Assessment. The quality appraisal of the studies included in this review was conducted after data extraction to reduce any reviewer bias from the author while extracting data [ 37 ]. As per the Centre for Reviews and Dissemination’s and Boland and scholars’ guidance, [ 37 , 38 ], the Joanna Briggs Institute’s (JBI’s) critical appraisal and bias assessment tools were chosen [ 41 , 42 , 43 ]. The JBI’s quality appraisal tools were selected due to the validation of these tools in healthcare settings as well as the fit of the tools for various study designs.

From the JBI, three critical appraisal tools were chosen to align with the study designs used in the studies included in this review. Each tool asks a series of closed-ended questions regarding research questions, methodology, data collection, representation, interpretation, reflexivity, and ethics [ 41 , 42 , 43 ]. The response options for each question included, “yes”, “no”, “unclear”, and “not applicable”, and for the purposes of this review, we added a response option of “partially” to account for some of the studies that met partial criteria of the question. First, for the qualitative studies, the JBI Critical Appraisal Checklist for Qualitative Research tool was applied to the five qualitative studies (see Supplement S1 ) [ 41 ]. For the quantitative studies, two different JBI critical appraisal tools were used to appraise the data of both cross-sectional and longitudinal study designs. The JBI Critical Appraisal Checklist for Cross-Sectional Studies tool was applied (see Supplement S2 ) [ 42 ], and for the two longitudinal studies included in the review, the JBI Critical Appraisal Checklist for Case Series (Longitudinal) Studies was applied (see Supplement S3 ) [ 43 ]. All of the studies generally met the quality appraisal checklist requirements, and a few studies are considered weaker studies due to lack of collection or reporting of demographic information and explicit naming and controlling for confounding variables in statistical analyses [ 16 , 18 , 19 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 ].

The systematic search initially produced a total of 248 records that were identified from 12 databases, 0 articles from registries, and 2 articles from consulting experts in the field (see Figure 2 ). Duplicate articles were removed from the records ( n = 104), leaving 146 records to be screened by title and abstract. After screening the 146 articles, 110 articles were excluded from the review for several reasons, including: keywords not being present in the title or abstract, no mention/measure of moral injury, no study of additional wellbeing measures, no association statistical analyses, and the articles were news articles, conceptual pieces, letters to the editor, or reviews. Finally, 36 articles were sought for retrieval, and 35 articles were successfully accessed. All 35 articles were assessed for eligibility, and 17 records were excluded for a variety of reasons, including: the article was a conceptual paper or review paper, had no measure of moral injury, used the wrong statistical analysis (not addressing the research question), or was a scale development paper (see Figure 2 ). The final group of articles retained in this review included 15 studies presented in 18 articles. Both quantitative ( n = 13) and qualitative papers ( n = 5) were included in order to understand both the occurrence and experience of moral injury, as this area of research is developing.

An external file that holds a picture, illustration, etc.
Object name is ijerph-20-06300-g002.jpg

PRISMA-P Consort Diagram.

3.1. Sample Characteristics

Of the fifteen studies identified, two studies did not collect or disclose demographic data [ 19 , 44 ]. One study did not report demographic information despite disclosing that they collected it [ 45 ]. Only six articles included in the review collected information on the racial identity of their participants, and all of these articles used quantitative methodology [ 47 , 51 , 52 , 53 , 57 , 58 ]. The majority of participants were of White or Asian ethnicity. Many different types of HCWs were included in the empirical studies in this review. Most of the participants were doctors and nurses, but other healthcare workers, psychologists, directors, and non-clinical staff were included in smaller numbers within larger physician and nursing samples [ 16 , 18 , 45 , 48 , 50 , 51 , 52 , 53 , 55 , 56 ]. Generally, the mean age was found to be between 20 and 41 years old across all studies except one study, which categorized the mean age to be under 55 years old [ 53 ]. Age was measured in both categorical and continuous formats in these studies.

Some of the studies focused on new HCWs, either in training programs within their first few years, or medical students [ 19 , 45 , 46 ], while other studies had a minimum years of experience requirement [ 18 ], with many participants often having more years of experience [ 16 , 44 , 49 , 51 , 52 , 53 , 57 , 58 ]. Some studies did not have a requirement for the number of years of work experience [ 47 , 48 , 50 , 54 , 55 , 56 ]. Across the studies, other variables were collected such as marital status, education degree, and religious affiliation. Finally, seven of the articles were based in the United States, three in the United Kingdom, two in mainland China, one in Australia, three in Israel, one in Turkey, and one study that sampled from both Austria and Italy.

3.2. Synthesis of Moral Injury and Wellbeing

Current empirical evidence suggests that moral injury is correlated with wellbeing outcomes for healthcare clinicians. Findings demonstrate that moral injury is positively correlated with wellbeing outcomes [ 16 ]. The association between moral injury and wellbeing are summarized in the two major subcategories reflecting the organization of the current literature: personal wellbeing and professional wellbeing. An overview of the descriptive, conceptual, and methodological findings is found below (see Table 1 ).

Definition of Moral Injury. In this review, three papers use Shay’s definition [ 11 ], ten papers use Litz and Kerig’s definition, and three papers reference both definitions. One paper references a systematic review on moral injury as the citation for moral injury [ 45 ], and this review cites both Shay’s and Litz and Kerig’s definition as well [ 11 , 12 , 59 ]. Last, one team used a less common definition by Koenig, Ames, and Nash [ 60 ].

3.3. Qualitative Studies Summary

The five qualitative studies included in this review used a variety of approaches to understand the experience of moral injury amongst healthcare workers. Three of the papers used a phenomenological design, in which semi-structured interviews and thematic analysis were used to explore the experience of moral injury as well as its impact on their wellbeing [ 19 , 45 , 46 ]. All three of the studies using phenomenology studied samples of new or early career healthcare workers [ 19 , 45 , 46 ]. The other two qualitative designs used grounded theory [ 18 ] and a case study [ 44 ], and both these studies looked at experienced HCWs experiences of moral injury. Kreh et al. used both individual semi-structured interviews as well as focus groups [ 18 ]. Alexander used chaplain case notes over three years with a physician experiencing moral injury (over 45 notes) for the case study [ 44 ]. Three of the papers used convenience sampling to recruit their participants [ 18 , 19 , 44 ], while the remaining two studies used purposive sampling [ 45 , 46 ]. Thematic analysis was used for the grounded theory and phenomenological studies [ 18 , 19 , 45 , 46 ], and content analysis was used for the case study [ 44 ].

Systematic Review Summary Table.

Authors Year	Aims	Sample (Size, Description, and Method)	Methodology/ Design/ Theory	Concepts Studied (Variables)	Outcomes	Conclusion about Moral Injury and Wellbeing
[ ]	To offer an illustration of how moral injury interventions with veteran populations can inform care for physicians experiencing burnout.	= 1 Female cardiologist with 20+ years clinical experience. Convenience Sampling	Qualitative Case Study Content Analysis No use of theory	Moral Injury (Shay’s Definition) Personal Wellbeing Compassion Fatigue Emotions Burnout	Themes: Use of clinical terms is not helpful in describing distress. Need to address the moral declination that impacts her personal wellbeing and work. Examination of all identities is essential. “Polarization” must be named in work vs. personal conflict.	Moral injury impacts personal wellbeing (adverse personal emotions, high stress, and polarization between work/personal life and beliefs) as well as professional wellbeing (burnout, compassion fatigue, and increased cynicism).
[ ]	To analyze these data with regard to positive and harmful ways trainees have been impacted by their clinical work.	= 14 Majority female ( = 11) psychologists in a medical center during the second year of their training program. Purposive Sampling	Qualitative Phenomenological Cross-sectional, semi-structured interviews Thematic Analysis [ ] Recommended Biopsychosocial-spiritual model for theory.	Moral Injury (MI) [ ] Vicarious Trauma (VT) Secondary Traumatic Stress (STS) Compassion Fatigue (CF) Burnout	Themes: Engagement with training and professional selves. Engagement with training and holistic selves. Self-Care	Trauma exposure could lead to STS, VT, and MI. MI can occur prior or alongside CF, and then burnout is a result of all these experiences.
[ ]	To examine the moderating role of thwarted belongingness in the relationships between HCWs’ exposure to potentially morally injurious events (PMIEs) and moral injury symptoms, depression, and anxiety.	296 Majority female, Israeli-born, and married. Mean age of 40.28 years, and included nurses, doctors, social and psychological care workers, and clinical support workers who mostly worked in hospitals. Convenience Sampling	Quantitative Cross-sectional Linger Regression Mediation-Moderation Modeling [ ] Lietz’s framework of moral injury named in discussion [ ].	Moral Injury Potentially Morally Injurious Events (PMIEs) Anxiety Depression Belongingness	Moral injury was positively correlated with anxiety, depression, PMIEs, and belongingness.	When healthcare workers are exposed to more PMIEs, they also experience moral injury symptoms, which is associated with anxiety and depression. The relation between PMIE and depression and anxiety is mediated via moral injury symptoms and moderated by thwarted belonging.
[ ]	To explore, “how do newly qualified doctors experience transition from medical school to practice” and “moral injury during transition”?	= 7 New doctors (first 4 years) with an age range of 24–29 years, predominantly female, and who recently experienced a transition (<2 years). Purposive Sampling	Qualitative Hermeneutic Phenomenology [ ]. Semi-Structured Interviews Thematic Analysis using an Interpretivist Paradigm [ , , ]. Multiple and Multidimensional Transitions (MMT) Theory [ ].	Moral Injury (Shay’s Definition) Transitional Experiences	Themes: The nature of transition to practice. The influence of community. The influence of personal beliefs and values. The impact of the unrealistic undergraduate experience.	Transition to practice was viewed negatively due to the lack of interpersonal support in 4-month rotations. Participants relied on the ethics of caring values to cope, but this in itself is troublesome and predisposes to moral injury.
[ ]	To examine the relationship between burnout, second victim experiences, and moral injury experiences before and during the COVID-19 pandemic among hospitalists.	81 Hospitalists between the ages of 20 and 40, with a stable partner/married, have children, and the majority of their work was clinical. Convenience Sampling	Quantitative Cross-sectional comparison Independent sample t-test No use of theory	Moral Injury Burnout Second Victim Experiences Well-being Work Wellbeing	Burnout levels reported were the same across pre COVID-19 and during COVID-19. An increase in reporting of second victim experiences during COVID-19, whether the hospitalist experiences burnout or not.	Moral injury was named as a predictive variable of burnout during COVID-19 in this study. During the pandemic, there was a higher rate of moral injury amongst burned out hospitalists.
[ ]	This study investigated the occurrence that HCPs were experiencing MI, whether the experience of MI was related to co-occurring psychiatric symptomatology, self/others MI, and burnout.	265 Majority white females with a mean age of 37.6 years old. Worked in a large city, have a college degree, and married or in a long-term relationship. Convenience Sampling	Quantitative Longitudinal Logistic Regressions Multilinear regression Multilevel modeling No use of theory	Moral Injury Healthcare Morally Distressing Experiences Current Psychiatric Symptomatology Workplace Burnout	Notably, longitudinally, self-moral injury was most impactful on experiences of burnout, and others moral injury was level influential on burnout. Higher levels of self-moral injury were correlated with higher levels of depression, anxiety, and PTSD, and other moral injury was only associated with depression.	When a healthcare worker conducts a moral injury themselves, they are most at risk for experiencing burnout. While witnessing others do things that healthcare workers find morally injurious can cause some depression, it is the individual moral injury that contributes to anxiety and PTSD.
[ ]	To develop basic hypotheses regarding resilience and stress experiences of healthcare workers in the first phase of the COVID-19 pandemic.	13 Healthcare workers (psychologists, physicians, and nurses) between the ages of 26 and 40, mostly female, with at least 5 or 10 years of experience for staff and clinicians, respectively. Convenience Sampling	Qualitative Grounded Theory Semi-Structured Interviews and Focus Groups Thematic Analysis [ ]. No use of theory	Moral Injury (Shay’s Definition; Litz’s Definition) Psychological Safety Stress Institutional Support Resilience	Themes: Fear, guilt feelings, frustration, loss of trust, and exhaustion Casual factors: rapidly evolving situations with high uncertainty Stressors Resilience factors 3 developed hypotheses	Stress, power imbalance, and inability to separate home from work were all named as precursors to moral injury. Then, moral injury could result in poor mental health.
[ ]	To highlight the emotional burden (depression and anxiety) among healthcare workers during COVID-19, and to further understand the direct and indirect role of PMIEs as well as the mediating role of stress and moral injury symptoms on depression and anxiety.	296 Majority female, Israeli-born, and married. Mean age of 40.28 years, and included nurses, doctors, social and psychological care workers, and clinical support workers who mostly worked in hospitals, with an average 12 years of experience. Convenience Sampling	Quantitative Cross-sectional Pearson’s Correlations Structured Equation Modeling No use of theory	Moral Injury (Litz’s Definition) PMIE Depression Anxiety Perceived Stress	PMIEs were significantly positively correlated with depression and anxiety. Stress and MI were also found to be significant mediating variables between PMIE and anxiety and depression. The full model explained 63% variance in depression and 57% variance in anxiety.	This study highlights the relationship between moral injury and stress as well as moral injury increasing anxiety and depression.
[ ]	To determine whether the concept of moral injury resonated with medical students working in emergency medicine and what might mitigate that injury for them.	= 5 Medical students in prehospital care. Convenience Sampling —Sampled using critical case sampling	Qualitative Phenomenological Structured Interviews Focus Groups Thematic Analysis [ ] No use of theory, names need for theoretical framing	Moral Injury (Shay’s Definition; Litz’s Definition) Trauma Exposure Social Support	Themes: What is Seen on Scene Material versus Human Resources The Complexity of Debrief	Moral injury acts as a reaction to witnessing trauma (but does not qualify as PTSD). Then, experiencing moral injury can lead to other wellbeing outcomes. Social supports and debriefing traumatic events are protective factors to reduce experiences of moral injury.
[ ]	The purpose of the project was to quantify experiences of moral injury anddistress in HCWs during the first three months of the COVID-19 pandemic response.	= 96 Majority female attending physicians with a mean age of 41 years old and an average of 14 years of experience in healthcare. Convenience Sampling	Quantitative Prospective Longitudinal Survey Design Descriptive Analysis Paired t-test Hierarchical Linear Modeling No use of theory	Moral Injury (Litz’s Definition) Resilience Distress	In the final model, stressful work environment was significantly associated with moral injury, while supportive work environment was nearly significantly associated with lower moral injury.	Stress and support are both related to moral injury, and stress was identified as a predictor to moral injury.
[ ]	To understand the extent to which healthcare workers experience moral injury while working in a pandemic.	109 Majority white, female physicians and nurses, with an average age of 38 years old and an average of 12 years in healthcare. Convenience Sampling	Quantitative Cross-Sectional Survey Design Descriptive, correlational, and Multiple regression analyses No use of theory	Moral Injury (Litz’s Definition) Professional Quality of Life -Compassion Satisfaction (CS) -Burnout -Secondary Traumatic Stress (STS)	STS was significantly associated with moral injury. Given the higher correlation between secondary traumatic stress and moral injury, a limited contribution of burnout was identified within the model, so burnout was removed.	STS was shown to significantly contribute to moral injury as a predictor. Burnout showed no association to moral injury, and CS was not significantly associated with moral injury.
[ ]	To (a) characterize the changes in HP moral injury wrought by the pandemic over the course of 2020 and (b) identify potential predictors of moral injury amongst HPs.	1831 Majority white, female, Christian, between the ages of 35–44, nurses and doctors, who are married. Snowball Sampling	Quantitative Cross-Sectional Survey Design Descriptive Student’s -test Pearson’s Correlations Logistic Regression No use of theory	Moral Injury (Shay’s Definition) Religiosity Burnout	Results indicated that significant negative predictors of MISS-HP included ages of more than 55 years old, greater religiosity, direct experience with patients with COVID-19, divorced, and non-nursing professions.	Moral injury is a parallel construct to burnout. Moral injury has been suggested as a precursor to burnout [ ], and it is possible that burnout rates will continue to increase as a lagging marker of ongoing moral strain. Personal identity factors impact moral injury.
[ ]	To a) examine the prevalence of moral injury symptoms causing impairments in family, social, or occupational functioningand b) identify predictors of MI symptoms in bivariate and multivariate analyses.	181 Majority white, male, physicians, with a majority of participants under the age of 55, who are Christian. Snowball Sampling	Quantitative Cross-Sectional Survey Design Descriptive ANOVA Student’s -test Pearson Correlations No use of theory	Moral Injury (Shay’s Definition; Litz’s Definition) Clinical Characteristics Religious Characteristics Depression Anxiety Burnout	Moral injury symptoms were significantly more common among individuals who were more depressed, who were more anxious, or, especially, who indicated more burnout symptoms. In the final model, the strongest predictor of MI symptoms was burnout, followed by commission of medical errors in the past month, and religiosity at a trend level.	Moral injury is correlated with individuals with higher rates of depression, anxiety, and burnout. Committing medical errors, younger age, lower religiosity, and fewer years in practice were all significant predictors of moral injury. Moral injury mediates the relationship between experiencing transgressing moral code and the clinical outcomes.
[ ]	To explore if healthcare providers in psychiatric settings are exposed to PMIEs, what the relationship between PMIEs and wellbeing are, and what the impact of COVID-19 is on PMIEs and wellbeing.	237 Majority of participants were female, white British, between 21 and 30, and unregistered nurses. Convenience Sampling	Quantitative Cross-sectional Survey Design Spearman Rank-Order Correlations Bootstrapped Regressions No use of theory	Moral Injury/PMIEs (Litz’s Definition) Wellbeing (Subscales: Burnout, Secondary Trauma, Compassion Satisfaction)	Moral injury has significant positive associations with burnout, secondary traumatic stress, and significant negative associations with compassion satisfaction.	Moral injury was predictive of higher secondary trauma and burnout as well as lower self-compassion amongst healthcare workers.
[ ]	To determine burnout levels and possible related psychological processes such as psychological flexibility, moral injury, and values among healthcare workers after the first year of the COVID-19 pandemic.	= 124 The sample was majority female doctors with a mean age of 33.3 years old. Convenience Sampling	Quantitative Cross-sectional Survey Design Correlation Analysis Multiple Linear Regression No use of theory	Moral Injury (Litz’s Definition) Psychological Flexibility Burnout Depression and Anxiety Values	Depression and anxiety were the only significant predictors of emotional exhaustion. Moral injury was the only significant predictor of depersonalization. Moral injury, days worked during COVID-19, and value obstruction were the significant predictors for personal accomplishment.	This study demonstrates associations between moral injury and burnout, specifically moral injury as a predictor of depersonalization and personal accomplishment within burnout.
[ ]	The objectives of this study were to examine patterns of exposure to potentially morally injurious events (PMIEs) among HSCWs and their associations with MI, mental health outcomes and psychological correlates.	296 Majority female, Israeli-born, and married, with a mean age of 40.28 years, and included nurses, doctors, social and psychological care workers, and clinical support workers who mostly worked in hospitals. Convenience Sampling	Quantitative Cross-sectional survey Design Latent Class Analysis No use of theory	Moral Injury Potentially Morally Injurious Events (PMIEs) Depression Self-Criticism Trauma Self-Compassion	Participants who had high exposure or betrayal exposure to moral injury experienced more PTSD and moral injury symptoms than those with minimal exposure. Those in the high exposure group also had more depressive symptoms. Additionally, those in the high exposure and betrayal only exposure groups had higher rates of self-criticism and lower self-compassion.	This study highlighted the relationship between moral injury and trauma (PTSD), mental health (depression), self-criticism and low self-compassion.
[ ]	To examine the relationship between spirituality, moral injury, and mental health among physicians and nursesin mainland China during the COVID-19 pandemic.	3006 Majority Han, female, doctors, with bachelor’s degree, married, and not affiliated with religion, with an average age of 35 years old, with an average of 12 years of practice. Snowball Sampling	Quantitative Cross-Sectional Survey Design Descriptive Pearson’s Correlations Students -test ANOVA Hierarchical Linear Modeling No use of theory	Moral Injury (Litz’s Definition) Spirituality Depression Anxiety	Spirituality was positively correlated with moral injury, depressive symptoms, and anxiety symptoms) after controlling sociodemographic variables.	Moral injury is correlated with mental illness. Those with higher spirituality were associated with experiencing higher moral injury. Moral injury a mediating variable but a moderating variable between spirituality and depression/anxiety.
[ ]	To assess the psychometric properties of the 10-item Moral Injury Symptoms Scale-Health Professional (MISS-HP) among healthcare professionals in China.	3006 Majority Han, female, doctors, with bachelor’s degree, married, and not affiliated with religion, with an average age of 35 years old, with an average of 12 years of practice. Snowball Sampling	Quantitative Cross-Sectional Survey Design Pearson’s Correlations Students -test ANOVA No use of theory	Moral Injury (Litz’s Definition) Spirituality Depression Anxiety Well-being Burnout	Moral injury had a small significant inverse correlation with personal accomplishment and a significant moderate inverse association with SFI. Otherwise, moral injury had a significant moderate positive association with the remaining constructs: PHQ-9, GAD-7, emotional exhaustion, and depersonalization.	Moral injury is found in increasingly stressed healthcare professionals, and moral injury is correlated with depression, anxiety, burnout (all three subconstructs), and flourishing.

3.4. Quantitative Studies Summary

Eleven studies used cross-sectional study design [ 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 ]. One study used a prospective longitudinal survey design [ 16 ], and another study used a case series longitudinal design [ 47 ]. Nine of the studies [ 16 , 47 , 48 , 49 , 50 , 51 , 54 , 55 , 56 ] used convenience sampling methods to recruit their participants and the remaining four quantitative studies used snowball sampling methods to recruit their participants [ 52 , 53 , 57 , 58 ]. Across the quantitative studies, the Moral Injury Events Scale (MIES) was used in six papers to measure moral injury [ 69 ], and the remaining four papers [ 52 , 53 , 57 , 58 ] used the Moral Injury Symptom Scale–Healthcare Provider (MISS-HP) [ 70 ], which was adapted from the Moral Injury Symptom Scale Military Short Form (MISS-M-SF) [ 60 ]. Three papers used both the MIES and the MISS-HP [ 48 , 50 , 56 ]. The MIES scale is a more generalized scale to measure moral injury, while the MISS-HP is a healthcare setting specific scale.

The quantitative studies used various measures and constructs to measure the concept of wellbeing amongst HCWs, accounting for both professional wellbeing outcomes and personal wellbeing outcomes [ 16 ]. The most common constructs that were analyzed in the quantitative articles were burnout, compassion fatigue/satisfaction, mental health, and spirituality/religiosity. For burnout, almost all of the tools used included an iteration of the Maslach Burnout Inventory (MBI) [ 53 , 55 , 57 ]; one study used the abbreviated MBI (aMBI) [ 52 ], another used the MBI–Human Services Survey for Medical Professionals (MBI-HSMP) [ 58 ]. When studying burnout, a few studies did not use an iteration of the MBI. Litam and Balkin [ 51 ], as well as Morris and scholars, instead used the Professional Quality of Life (ProQOL) with a subscale of burnout [ 54 ], while Chandrabhatla and scholars used the Mini Z burnout survey [ 49 ], and the Professional Fulfillment Index was also used [ 47 ]. For compassion fatigue/satisfaction, Litam and Balkin and Morris and scholars also analyzed this construct using the ProQOL scale with a subscale of compassion satisfaction [ 51 , 54 ]. The other studies that analyzed compassion fatigue/satisfaction were qualitative. A variety of scales were used to measure spirituality/religiosity. All of the scales used were validated and reliable. The scales used in the studies to measure spirituality/religiosity included the Duke University Religion Index (DUREL) [ 52 ], the Belief into Action Scale (BIAC) [ 53 ], and visual analogue scales [ 57 , 58 ]. Last, mental health was measured through secondary traumatic stress, depression, and anxiety, using the ProQOL [ 51 ], PHQ-9, GAD-7 [ 47 , 48 , 50 , 53 , 56 , 57 , 58 ], the Global Mental Health–K6 Scale [ 56 ], and through the Depression Anxiety Stress Scale [ 55 ].

3.5. Personal Wellbeing

The following constructs were used to measure personal wellbeing: “personal wellbeing” [ 44 , 58 ], emotions [ 44 ], transitional experiences [ 46 ], stress/distress [ 16 , 18 , 47 , 50 , 56 ], resilience [ 16 , 18 , 47 ], spirituality/religiosity [ 52 , 53 , 57 , 58 ], psychological safety [ 18 ], social support [ 19 ], thwarted belonging [ 48 ], flourishing [ 49 ], life satisfaction [ 49 ], psychological flexibility [ 55 ], self-criticism [ 56 ], self-compassion [ 56 ], valuing [ 55 ], and mental illness, including both depression and anxiety [ 48 , 50 , 53 , 55 , 57 , 58 ].

All studies found an association between moral injury and personal wellbeing. One qualitative study found that moral injury impacted personal wellbeing, specifically increasing stress, emotions, and polarization between personal and work life [ 44 ]. Other studies found that adverse personal wellbeing is a risk factor for experiencing moral injury [ 16 , 46 , 53 ]. For example, Brown and scholars [ 46 ] found that the nature of transitional experiences can cause disruptions in physician wellbeing, which could then lead to moral injury. Other studies named moral injury as a mediating variable between multiple personal wellbeing outcomes [ 18 , 19 , 48 , 50 , 57 , 58 ]; such as, when a healthcare worker experiences stress, they can then experience moral injury, which could lead to adverse mental health outcomes [ 18 , 48 , 50 ]. Interestingly, resilience was not associated with moral injury across these studies [ 16 , 18 , 47 ]; yet, Zerach and Levi-Belz demonstrated a relationship between moral injury and self-criticism and low self-compassion [ 56 ].

3.6. Professional Wellbeing

The studies that measured professional wellbeing included these constructs: compassion fatigue/satisfaction [ 44 , 45 , 51 ]; trauma exposure [ 19 , 47 ]; vicarious trauma [ 45 ]; secondary traumatic stress [ 45 , 49 , 51 , 54 ]; burnout [ 44 , 45 , 47 , 49 , 51 , 52 , 53 , 54 , 55 , 58 ]; and institutional support [ 18 ]. Burnout was found to be both associated [ 19 , 45 , 52 , 53 , 58 ] and not associated with moral injury [ 51 ]. Of those studies that found that moral injury and burnout are associated, some studies described burnout as an outcome of experiencing moral injury [ 19 , 45 , 53 , 58 ]; alternatively, one described burnout as a parallel construct to moral injury, where the constructs impact one another, but they co-exist [ 52 ], and another two studies found burnout to be a predictor of moral injury [ 49 , 55 ]. In three of the studies included in this review, compassion fatigue was found to be an outcome of moral injury [ 44 , 45 , 54 ], and in another study, was found not to be significantly related to moral injury [ 51 ].

Trauma exposure, across the studies included in this review, was measured using three different types of trauma (trauma exposure, vicarious trauma, and secondary traumatic stress). All three forms of trauma were framed as predictors of moral injury [ 45 , 51 ]. Multiple studies that identified trauma, or more specifically a trauma-response, as an outcome of moral injury named the clinical diagnosis or PTSD, which does not always occur after experiencing moral injury [ 19 , 47 , 53 , 56 ]. Institutional support was named as a protective factor for moral injury, and power imbalances were identified as a risk factor for experiencing moral injury [ 18 ].

3.7. Use of Theory

Out of the 18 articles included in this study, one paper included a specified theory [ 46 ]. The qualitative study that used this theory aimed to answer a question regarding how newly qualified doctors experience the transition from medical school to practice [ 46 ]. The authors of this paper suggested the use of Multiple and Multidimensional Transitions (MMT) Theory [ 67 ]. The theory in this study was used to contextualize the transitional time in which the participants could have higher levels of vulnerability rather than directly using theory to frame moral injury [ 46 ]. Two additional articles acknowledged the need for theoretical framing when studying moral injury amongst HCWs [ 19 , 45 ]. Ball and scholars [ 45 ] named the biopsychosocial–spiritual model [ 71 ] as a potential theory to contextualize moral injury in healthcare. Murray and scholars acknowledged the need for theoretical framing, but they did not recommend any specific theories or frameworks [ 19 ].

4. Discussion

This systematic review identified 18 original empirical articles that examined the relationship between moral injury and personal and professional wellbeing amongst HCWs. This review found that there is a direct link in the literature between moral injury and wellbeing amongst HCWs. An association between moral injury and wellbeing was identified across both personal and professional wellbeing, but the temporal order of moral injury and wellbeing-related outcomes remains unclear. Across these studies, there were relationships found between moral injury and related constructs including burnout, trauma (vicarious trauma and secondary traumatic stress), compassion fatigue, mental health (depression, anxiety, and PTSD), and stress. Since this review identifies associations between moral injury and several wellbeing outcomes, it is demonstrating the vast impact that the experience of moral injury has on HCWs.

4.1. Theoretical Framing

As found in this review, there was a lack of theoretical orientation in the studies, and this reflects the early stage of moral injury research in healthcare. The studies identified in this review are essential in contributing to the theoretical framing of moral injury in healthcare as they provide information on the phenomenon of moral injury and its association to other related constructs of wellbeing. However, these studies do not conceptually illuminate how personal and professional identities contribute to the experience of moral injury. Even further, a commonly named risk factor for moral injury has been identified as power imbalance (systemically and interpersonally); yet, power was not measured across any of the studies in this review. This issue could be addressed by using existing theoretical frames to articulate the role of power/imbalance in the experience of moral injury. Specifically, using systems-level theory aligns with the concept of moral injury, as moral injury speaks to systems-level causes and solutions as opposed to burnout, which generally alludes to individual level causes, symptoms, and solutions [ 72 , 73 , 74 ].

4.2. Power as a Measured Construct

In the study of moral injury in healthcare, there remain numerous gaps in understanding the true nature of moral injury and how to intervene. First, in the study of moral injury, power is an essential asset to consider. Often, moral injury occurs when a power imbalance occurs between two or more people [ 11 ]. For example, if a supervisor requires a staff member to complete a task that the staff member disagrees with, the supervisor has the power to enforce the task completion, and the staff can in turn experience moral injury.

4.3. Consequences of Moral Injury for Healthcare Workers

The experience of moral injury captures a nuanced response to the challenges that HCWs face daily. HCWs, specifically doctors and nurses included in this review, work with people experiencing complex health demands, and the solution-searching in the midst of crisis that is required of HCWs is demanding. When a HCW experiences a moral injury, they can experience a variety of known and unknown outcomes. Studies included in this review demonstrate that experiencing moral injury causes emotional consequences of guilt and shame as well as values consequences of internal confusion with oneself. A HCW experiencing guilt and shame surrounding their work while caring for patients is having a juxtaposed experience. It is challenging for HCWs to be in the spotlight of implementing healthy practices and policies when they themselves are feeling upset or confused about the decisions they have made or witnessed within healthcare systems. These consequences of moral injury could contribute to HCWs feeling lonely, retreating from social and institutional support, and in turn leaving healthcare. Yet, within the current study of moral injury, researchers have found that naming moral injury as an experience that healthcare workers may face reduces a sense of isolation, feelings of guilt and shame [ 70 , 75 ].

4.4. Contextualizing Moral Injury within Wellbeing

Essential components contributing to the greater context of HCW moral injury and wellbeing are pay inequity, high caseloads, crisis standard of care, staffing shortages, HCW abuse (from the system and the patients), and the privatization and profiting of healthcare services in capitalistic countries. It is important to note that with the rapidly emerging research on moral injury in healthcare, there are still many unknown consequences of moral injury. This review serves as a beginning insight into opportunities for further understanding the consequences of moral injury within pre-existing wellbeing indices. Because the main constructs found to be associated with moral injury were burnout, trauma, mental health, and spirituality/religiosity, each of these will be discussed in turn.

4.5. Burnout

Previous literature has continuously highlighted the experience of burnout as the predominant way in which professional wellbeing was measured in healthcare settings [ 72 , 76 ]. Professionally, wellbeing literature often debates the constructs of burnout and moral injury, and scholars either distinctly separate these two terms or use them interchangeably [ 77 ]. This review demonstrated no collinearity between the constructs, although they are indeed associated, thus confirming the differences between burnout and moral injury. Conceptually, the emotional consequences of moral injury align with the domain of emotional exhaustion within burnout [ 55 ], identifying a potential pathway of the relationship between moral injury and burnout. Another sub-construct of burnout is depersonalization, meaning feeling unlike oneself, and this is conceptually related to the consequence of internal confusion about oneself from moral injury.

4.6. Trauma

Trauma via multiple mechanisms, including secondary/vicarious trauma and primary trauma exposure, were found to be associated with moral injury, demonstrating the relevance of trauma in the experience of moral injury. Moral injury, as a phenomenon, is specific to high stakes situations. Healthcare, innately, is a high stakes environment, and often HCWs are exposed to high levels of trauma compared to the general public [ 54 ].

4.7. Mental Health

Moral injury and mental health demonstrated a consistent association across articles included in this review. When referring to mental health, this includes all diagnosable mental health disorders, but often arose as depression, anxiety, and trauma responses. In a recent scoping review of moral injury, a scholar listed primary and secondary consequences of moral injury, and they state that depression, anxiety, and self-harm are all potential symptoms caused by moral injury [ 75 ]. With the high rates of mental health diagnoses amongst HCWs in general [ 78 ], the current review contextualizes moral injury’s role alongside mental health. Specifically, this review did not measure for PTSD or acute stress disorder (ASD). While all moral injury experiences may not lead to PTSD or ASD, it is important to account for mental health diagnoses pertinent to trauma as trauma was repeatedly associated with moral injury across the articles in the current study.

4.8. Spirituality/Religiosity

Spirituality/religiosity demonstrated a strong association with moral injury across articles. Often individuals develop their moral orientation from their environment and systems they belong to, and historically, morals are often taught in systems of religion and spirituality (i.e., temples or churches). This association provides some insight into how some individual beliefs influence their experiences of moral injury. For example, what may feel right or wrong to one person may differ from the next person based on their belief system. Using spirituality/religiosity is one mechanism of measuring individual belief systems.

4.9. Measuring Moral Injury

This review also demonstrates the exploratory nature of studying moral injury in healthcare settings through sampling and data collection methods. Across qualitative and quantitative methods, convenience sampling was predominantly used. Qualitatively, the research questions focused on understanding the phenomenon of moral injury amongst HCWs. Quantitatively, most of the studies used cross-sectional data collection methods that supported a general understanding of moral injury through surveys and scales. The study of moral injury amongst HCWs is a concept that is in a foundational research stage.

4.10. Synthesis of the Literature

The articles included in this review explored the relationship between moral injury and wellbeing amongst healthcare workers around the globe. Both qualitative and quantitative articles were included in this review, and each methodology provided unique insights. Specifically, the quantitative articles presented rates and severity of moral injury and additional wellbeing outcomes, highlighting the significant relationships between moral injury and several additional measures of wellbeing. The statistical representation of this association is valuable; yet, the qualitative studies generally provided more complex, unique, and deep understanding on moral injury and wellbeing. The qualitative studies were able to identify causes, potential mediators between moral injury and wellbeing (e.g., material versus human resources [ 19 ]), complex emotional consequences of moral injury (e.g., fear and frustration [ 18 ]), and highlight imaginative solutions (e.g., debriefing sessions [ 19 ]) to ameliorate moral injury. Additionally, the qualitative studies’ samples differed from the quantitative samples, as the qualitative samples explicitly included or targeted students and trainees in their studies, whereas the quantitative studies did not explicitly name the inclusion of trainees in their samples.

Furthermore, a few differences were noted when comparing U.S.-based studies to studies from other countries. First, similar sample differences were noted when comparing the United States to studies from other countries in this review as noted when comparing the qualitative to quantitative studies. The U.S.-based studies did not explicitly include students and trainees, where there was more inclusion of these groups around the world. Moreso, the samples in the United States predominantly included physicians and nurses, whereas non-U.S.-based studies included a wider range of health professionals, including psychologists, social and psychological care workers, and clinical support workers, acknowledging the need to support the entire healthcare workforce. Last, it is important to note that non-U.S.-based studies included more psychosocial measures of wellbeing beyond the traditional understanding of professional and personal wellbeing. Other countries included measures of belonging, social support, psychological safety, psychological flexibility, and values-based questions, which add to the depth of understanding of moral injury and wellbeing amongst healthcare workers. In all, while there are differences across countries, samples, and methodologies found in this review, this work speaks as a cohesive body, offering a significant insight into the impact moral injury has on HCWs’ wellbeing.

4.11. Future Directions in Research

In forefronting power dynamics in the study of moral injury, scholars can be more inclusive of who is represented in their study samples, recognizing that other healthcare staff (i.e., social workers, housekeeping staff, certified nursing assistants) are at risk for experiencing moral injury due to their lack of power within the healthcare system. Next research steps should include the study of social workers and mental health workers in the understanding of moral injury in healthcare. Specifically, the phenomenon of moral injury should be better understood from the ground up as it uniquely impacts professions differently. Future moral injury exploratory research should also include the measure of power/imbalance, as empirical support on the role of power in the experience of moral injury is needed. When studying power/imbalance, the impact of social support and collaboration on moral injury and wellbeing should be examined.

In shifting from exploration to intervention research, much work is to be completed. Current interventions on moral injury are individual interventions; however, when power is named as a tenet or assumption of moral injury, aligned interventions could, and should, be directed at the systemic level. Addressing moral injury at the systemic level is essential, and moral injury healthcare research has not studied the outcomes of systems-level interventions. Current literature has tested a few interventions for healing the experience of moral injury, such as using acceptance and commitment therapy and hosting moral rounds or lunches at work [ 79 , 80 ].

5. Limitations

This review is not without limitations. Moral injury, which is not always named as “moral injury”, is a concept that is reported in articles beyond the ones included in this review under varying additional terminology (e.g., moral distress, moral emotion, and moral wrongdoing). For example, articles that used the term moral distress were excluded from this review, and while that provided a specific scope of research when studying the association of moral injury and wellbeing, moral distress and moral injury are terms that are often conflated in the literature, and studies may have been excluded that would otherwise fit the scope of this research. Moreso, moral injury is a concept which is recently gaining traction, and there are multiple definitions used for this term. Moral injury in this review was not limited to one definition of the term [ 11 , 12 ], and within the literature reviewed, multiple different scales to measure moral injury were used, making comparisons of rates of moral injury and associations to other wellbeing outcomes less consistently reliable. Additionally, this review used broad inclusion criteria for the term wellbeing, including wellbeing across professional and personal domains. While broad inclusion criteria on wellbeing were essential due to the emerging nature of moral injury in healthcare research, it also may have provided too many associations of moral injury and wellbeing to consider. Further, although a strength of this review was that two coders followed the review protocol, selection bias may have still occurred when selecting articles due to our positionalities. Last, the generalizability of this review is limited as several of the articles included in this review are based in the United States. The United States’ healthcare system fundamentally operates and is valued differently than in the majority of the world. Specifically, the capitalistic frame of healthcare services in the US, the US policy implemented for public health crises, and the lack of respect for the healthcare workforce all contribute to wellbeing in uniquely distinct ways than in other countries.

6. Conclusions

These studies, and their associated methodologies, each contribute to the greater meaning of moral injury in healthcare. This review begins to transition research from exploration and understanding to association and trends, in synthesizing the connection of moral injury to other wellbeing constructs in the field of healthcare. In the context of moral injury, its association with trauma is unsurprising; yet, a deeper understanding of the nature of the relationship between trauma and moral injury is urgently necessary. Future research should seek to incorporate additional tools that measure individuals’ belief systems in the study of moral injury to gain a better understanding of how non-personal beliefs are associated with moral injury. Additionally, future research should study moral injury across the allied health professions. This review makes the first steps in this identification process and in addressing gaps in the existing interventions, and begins to thread together rates, narratives, and conceptual framing connecting moral injury, burnout, and mental health outcomes. Future research should include intervention research to help identify strategies to ameliorate the experiences of moral injury and its associated outcomes.

Acknowledgments

Pari Thibodeau would like to acknowledge her dissertation committee members, Jennifer Greenfield, Heather Taussig, Karen Albright, and Michael Talamantes for the mentorship they provided on this paper within her dissertation.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph20136300/s1 , Supplement S1: Joanna Briggs Institute Critical Appraisal Checklist for Qualitative Research. Supplement S2: Joanna Briggs Institute Critical Appraisal Checklist Case Series (Longitudinal) Studies. Supplement S3: Joanna Briggs Institute Critical Appraisal Checklist for Cross-Sectional Studies.

Funding Statement

This research received no external funding.

Author Contributions

P.S.T. contributed to this paper in conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing—original draft preparation, writing—review and editing, visualization, supervision, and project administration. A.N. contributed to this paper in formal analysis, investigation, data curation, writing- review and editing. J.C.G. contributed to this paper in writing—review and editing. J.L.B. contributed to this paper in conceptualization, methodology, software, resources, writing- review and editing, supervisions, and project administration. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

COMMENTS

Public and patient involvement in quantitative health research: A
The majority of studies included in recent reviews of impact for public and patient involvement (PPI) in health research had a qualitative design. PPI in solely quantitative designs is underexplored, particularly its impact on statistical analysis. Statisticians ...
Recent quantitative research on determinants of health in high income
Background Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature. Methods We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019).
A Quantitative Observational Study of Physician Influence on Hospital
A Quantitative Observational Study of Physician Influence on Hospital Costs. Herbert Wong, ... Specific physician characteristics influence a physician's practice style as well as health care cost, delivery of care and outcomes. ... and/or publication of this article: This study was funded by the Agency for Healthcare Research and Quality ...
A review of the quantitative effectiveness evidence synthesis methods
The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development (including meta-analysis) for dealing with heterogeneity of intervention ...
Quantitative Results of a National Intervention to Prevent Hospital
Objective: To evaluate the effect of a multimodal initiative on CAUTI in hospitals with high burden of health care-associated infection (HAI). Design: Prospective, national, nonrandomized, clustered, externally facilitated, pre-post observational quality improvement initiative, for 3 cohorts active between November 2016 and May 2018.
Living with a chronic disease: A quantitative study of the views of
A pilot study was conducted with in-depth interviews with three patients with different diagnoses. The results of the interviews were analysed by the research group to identify crucial experiences of having a chronic disease. The research group then constructed the questionnaires based on the results.
A review of the quantitative effectiveness evidence ...
The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development (including meta-analysis) for dealing with heterogeneity of intervention effects, inappropriate 'lumping' of interventions, adjusting for different populations and outcomes and the inclusion of various ...
Quantitative research: Designs relevant to nursing and healthcare
This paper gives an overview of the main quantitative research designs relevant to nursing and healthcare. It outlines some strengths and weaknesses of the designs, provides examples to illustrate the different designs and examines some of the relevant statistical concepts.
A quantitative study of nurses perception to advance directive in
The study evaluated nurses' perceptions on the benefits, drawbacks, and their roles in initiating and implementing advance directives (AD) at private and public secondary healthcare units. The study adopted a cross-sectional, comparative-descriptive research design and was anchored on the structural functional theory. A total of 401 nurses (131 private and 270 public) were chosen on purpose.
Quantitative measures of health policy implementation determinants and
Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized.
Review Article Synthesizing Quantitative Evidence for Evidence-based
The purpose of this paper is to introduce an overview of the fundamental knowledge, principals and processes in SR. The focus of this paper is on SR especially for the synthesis of quantitative data from primary research studies that examines the effectiveness of healthcare interventions. To activate evidence-based nursing care in various ...
Quality in health care: possibilities and limitations of quantitative
In health care quantitative methodology is often used to address these quality dimensions and introduce patients' views and opinions. There are various benefits using quantitative research instruments, such as a detailed analysis of the importance of various quality dimensions for patients and an analysis of factors influencing patients ...
Barriers to help-seeking, accessing and providing mental health support
This study used a mixed methods sequential design, consisting of two distinct work packages: (1) quantitative survey to describe patterns of help seeking and unmet mental health needs and (2) nested semi-structured interviews to understand more nuanced aspects of accessing and delivering support.
A quantitative systematic review of the association between nurse skill
To examine the association between nurse skill mix (the proportion of total hours provided by Registered Nurses) and patient outcomes in acute care hospitals.A quantitative systematic review included studies published in English between January 2000 - ...
Using quantitative and qualitative data in health services research
In this methodological paper we document the interpretation of a mixed methods study and outline an approach to dealing with apparent discrepancies between qualitative and quantitative research data in a pilot study evaluating whether welfare rights advice has an impact on health and social outcomes among a population aged 60 and over.
Quantitative health impact assessment: current practice and future
Study objective: To assess what methods are used in quantitative health impact assessment (HIA), and to identify areas for future research and development. Design: HIA reports were assessed for (1) methods used to quantify effects of policy on determinants of health (exposure impact assessment) and (2) methods used to quantify health outcomes resulting from changes in exposure to determinants ...
Assessing the Outputs, Outcomes, and Impacts of Science Communication
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded by the Swiss National Science Foundation (SNSF).
Recent quantitative research on determinants of health in high income
The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries. Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art.
Influences on patient satisfaction in healthcare centers: a semi
Knowledge of ambulatory patients' satisfaction with clinic visits help improve communication and delivery of healthcare. The goal was to examine patient satisfaction in a primary care setting, identify how selected patient and physician setting and characteristics affected satisfaction, and determine if feedback provided to medical directors over time impacted patient satisfaction.
Patient-reported outcomes and experiences of migrants enrolled in a
Background Scholars recommend providing migrants living with HIV (MLWH) with free treatment, rapidly, once linked to care to optimize their HIV-related experiences and health outcomes. Quantitative evaluations of patient-reported measures for MLWH in such models are necessary to explore the viability of these recommendations. Methods Within a 96-week prospective cohort study at a ...
Appraising Quantitative Research in Health Education: Guidelines for
This publication is designed to help provide practicing health educators with basic tools helpful to facilitate a better understanding of quantitative research. This article describes the major components—title, introduction, methods, analyses, results and discussion sections—of quantitative research. Readers will be introduced to ...
Modeling factors affecting the design and manufacture of the
Dr Ramin Ravangard is a full professor in the field of healthcare services management and a member of the Healthcare Services Management Department and the Human Resources Research Center at the Medical Sciences University. He has collaborated with Dr Mohammadkarim Bahadori on research projects for many years.
Diagnostic value of aMMP‐8 and azurocidin in peri‐implant sulcular
Objective. The objective of this study was to investigate the effectiveness of testing for active matrix metalloproteinase-8 (aMMP-8) by a quantitative point-of-care (PoC), chairside lateral flow immunotest and azurocidin, in the peri-implant sulcular fluid (PISF), as biomarkers for the presence or absence of peri-implant diseases.
PLOS Pathogens
PLOS Pathogens is a peer-reviewed journal that publishes cutting-edge research on the biology and impact of pathogens. Explore diverse topics such as peroxisomes, coral bleaching, and antibiotic resistance.
PDF Communities Advancing Research Equity for Health (CARE for Health™)
Research Equity for Health (CARE for Health™) Conduct research addressing . issues important to diverse ... Integrate research into the clinical care environment . Engender trust in science by addressing community needs Achieve longitudinal collection of clinical data to address health across the lifespan Reduce burden on providers using ...
Synthesising quantitative and qualitative evidence to inform guidelines
Guideline developers are increasingly dealing with more difficult decisions concerning whether to recommend complex interventions in complex and highly variable health systems. There is greater recognition that both quantitative and qualitative evidence ...
What Is Data Analysis? (With Examples)
Descriptive analysis tells us what happened. This type of analysis helps describe or summarize quantitative data by presenting statistics. For example, descriptive statistical analysis could show the distribution of sales across a group of employees and the average sales figure per employee.
Quantitative research on the impact of COVID‐19 on frontline nursing
The aim of the study was to examine the relationship between stress, psychological symptoms and job satisfaction among frontline nursing staff at a military hospital in Saudi Arabia during the COVID‐19 pandemic.Descriptive cross‐sectional ...
Inclusive Alliance Wins National Grant to Enhance its Community Care
Quantitative strategies, Wall Street-caliber research, and insightful market analysis since 1998. Recent Quotes. My Watchlist ... infrastructure funding to the selected CCHs like Inclusive Alliance in support of their contracting efforts with health care organizations for coordinated access to and delivery of social care programs and services ...
The Association of Moral Injury and Healthcare Clinicians' Wellbeing: A
Both qualitative and quantitative articles were included in this review, and each methodology provided unique insights. Specifically, the quantitative articles presented rates and severity of moral injury and additional wellbeing outcomes, highlighting the significant relationships between moral injury and several additional measures of wellbeing.

Save citation to file

Add to My Bibliography

Quantitative Results of a National Intervention to Prevent Hospital-Acquired Catheter-Associated Urinary Tract Infection: A Pre-Post Observational Study

Similar articles

Publication types

Related information

Miscellaneous

A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines

Conclusions

Research questions

Update of NICE public health intervention guidelines review

Background to the previous review

Search results and types of reviews

Comparison of review types to original review

Characteristics of guidelines using meta-analytic methods

Reasons for not using meta-analytic methods

Comparison of meta-analysis characteristics to original review

Application of evidence synthesis methodology in a public health intervention: motivating example

Evaluating the effect of interventions for promoting the safe storage of cleaning products to prevent childhood poisoning accidents

Multiple interventions: lumping or splitting?

Statistical heterogeneity

Tailored decision making to specific sub-groups

Further methods / other outcomes

Software and guidelines

Availability of data and materials

Abbreviations

Acknowledgements

Author information

Contributions

Corresponding author

Ethics declarations

Consent for publication

Additional information

Supplementary Information

Additional file 2

Rights and permissions

About this article

Share this article

BMC Public Health

A quantitative study of nurses perception to advance directive in selected private and public secondary healthcare facilities in Ibadan, Nigeria

Introduction

Theoretical orientation

Research design

Participants

Study location

Sampling technique

Research instrument

Data analysis

Ethical consideration

Characteristic and representative of nurses in the study

Distribution of nurses perception of benefits of advance directive

Perceived nurses role in advance directives

Perceived shortcomings of advance directive

Classification of nurses responses into positive and negative perception

Differences on nurses perception to advance directive

Availability of data and materials

Acknowledgements

Author information

Contributions

Corresponding author

Ethics declarations

Consent for publication

Competing interests

Additional information

Rights and permissions

About this article

Share this article

BMC Medical Ethics

Quantitative measures of health policy implementation determinants and outcomes: a systematic review

Conclusions

Registration

Contributions to the literature

Inclusion and exclusion criteria

Screening procedures

Data extraction strategy

Quality assessment of measures

Data synthesis and presentation

Implementation outcomes

Determinants of implementation

Tables of measures