mixed methods research reporting guidelines

Human Subjects Office

August 2024 irb connection newsletter, tips for web-based survey research platforms, student pi training requirement: reminder to faculty advisors and student pis, assistance for new faculty and research scientists: research navigation tool, icts clinical lecture series: working with the irb, updates for the ui research community, in the news, irb educational resources.

The Human Subjects Office (HSO) has published a new educational tool focused on two of the web-based survey research platforms that are available to faculty, staff, and students at the University of Iowa. The publication provides tips on using Qualtrics and REDCap, including the different features available on the platforms, as well as tools for avoiding common issues, such as over-enrollment, multiple survey submissions, and bot activity.  

Topics addressed in the educational tool include:  

survey testing  

survey editing  

creating logic quotas and  

bot detection  

institutional policies related to survey-based research  

training and other support available   

The new educational tool is available on the HSO website: Survey Based Research Educational Tool    

Student PI Training Requirement: Reminder to Faculty Advisors and Student PIs  

Graduate and undergraduate student principal investigators (PIs) are required to view recorded trainings about the HawkIRB New Project form (Parts 1 and 2) prior to submitting an application in HawkIRB. While the training is required for student PIs, it is recommended for all new faculty, staff, residents, fellows, professional students, and HawkIRB Delegates who will be using the HawkIRB system.  

The student PI training policy was instituted to improve the quality of HawkIRB submissions and ensure efficiency of the IRB review process for all IRB applications. The training, available in the IRB ICON Course for Researchers , will assist student PIs in navigating the HawkIRB system and preparing thorough HawkIRB applications. Note to Faculty advisors and others who work with student PIs : Please make sure graduate and undergraduate student PIs know to complete this training.  

More information is available about the training requirement at Roles and Responsibilities / Role of the Student PI , and more detailed information in the September 2022 IRB Connection Newsletter article .   

Other educational resources are also available on the Human Subjects Office website :  

IRB Overview Recording – If you did not hear this in a research methods course, you can access it in the IRB ICON Course for Researchers .  

Do I Need IRB Approval – if you are uncertain whether your project is human subjects research requiring approval from the UI IRB ( Get Started / Do I need IRB Approval )  

Exempt Status – Most social/behavioral research qualifies for Exempt Status. You can read about this in the Exemption Tool (under Get Help / Educational Tools ) and learn about it in the Part 5 HawkIRB training in the IRB ICON Course for Researchers .  

HawkIRB Submissions – There is additional information about the IRB Review Process, HawkIRB System and New Project forms on the Submit to IRB web page.  

Assistance for New Faculty and Research Scientist s : Research Navigation Tool   

By emily shultz, cip  .

UI Research Administration units use a Research Navigation Tool to begin the onboarding process for new UI faculty and research scientists. This tool provides information about a wide range of UI programs that provide support for research. This tool collects information about the nature and type of research you conduct to help us better assist you with setting up your research program.  

Click here to use the Research Navigation Tool  

On a monthly basis, HSO staff sends an email with a link to the tool to all new faculty and research scientists. Based on research plans indicated in the Research Navigation Tool, representatives from the following units contact the researcher to provide guidance and support for transferring or initiating research activities:   

Office of the Vice President for Research (OVPR)  

Research Development Office (RDO)  

Division of Sponsored Programs (DSP) (work with the departmental grant administrator)

Human Subjects Office (HSO) and Institutional Review Board (IRB)  

Offices of Animal Resources (OAR) and the Institutional Care and Use Committee (IACUC)  

Environmental Health & Safety (EHS)  

University of Iowa Research Foundation (UIRF)  

Institute for Clinical and Translational Science (ICTS)  

Information Technology Systems (ITS)  

For researchers who work with human participants, the Human Subjects Office offers a one-on-one IRB orientation to provide guidance, resources and support. This session provides information about:   

The Human Research Protection Program (HRPP)  

The IRB review process and requirements for submission  

Other committees or entities that review and approve human subjects research applications  

Features of the eResearch application system (HawkIRB), including the delegate permission system  

Educational resources available for all human subjects researchers  

Contact the IRB Education & Outreach team to learn about other educational resources for faculty and research scientists.  

ICTS Clinical Lecture Series: Working WITH the IRB  

By kelly o’berry , bs, cip and deb o’connell-moore, mba  .

At the June Clinical Lecture Series hosted by the Institute for Clinical and Translational Science (ICTS), Kelly O’Berry and Deb O’Connell-Moore provided an overview of the working relationships between researchers and the Institutional Review Board (IRB) / Human Subjects Office (HSO).   

Novel Research Methods  

Researchers are encouraged to consult with colleagues in their college or department to see if others have received IRB approval for similar procedures, as their insights and experience could be helpful. Researchers can also consult with HSO staff and IRB Chairs about novel research methods. An advance consult can help the HawkIRB application go through the IRB review process more smoothly.  

Ask Questions  

HSO staff invite researchers to ask questions and seek guidance at any time, but especially prior to submission of forms in HawkIRB. HSO staff can be reached phone (319-541-0326), email ( [email protected] ) or at IRB Office Hours via Zoom .   

The ICTS Regulatory Core is also available to assist with their virtual walk-in hours for guidance and support .  

Request Clarification  

During the IRB review process, HSO staff typically communicate with researchers through HawkIRB Workflow. This satisfies a regulatory requirement to maintain records of communications between the IRB and the PI. Researchers are encouraged to ask for clarification about a Workflow question/request if they are unsure of how to best address the revision.   

To review a recording of this ICTS Clinical Lecture Series presentation, go to the IRB ICON Course for Researchers .  

Updates for the UI Research Community  

Changes Coming to ClinicalTrials.Gov Platform  

(link sends e-mai

• Dementia Risk Factors Identified in New Global Report Are All Preventable - Addressing Them Could Reduce Dementia Rates by 45% - The Conversation  
• Black employees more likely to experience depression after workplace mistreatment – Newswise  
• First NonInvasive Method to Continually Measure True Blood Pressure – Science Daily  
• Parents' eating behavior influences how their children respond to food – Science Daily  
• Blood Test Uses 'protein clock' to predict Risk of Alzheimer's and other diseases – Nature  

IRB Educational Resources   

Irb presentation : the roadmap to regulatory compliance: guidelines for clinicaltrials.gov registration & results reporting  .

This presentation will clarify the key aspects of ClinicalTrials.gov registration and results reporting, with an overview of guidelines and practical tips to avoid common errors that researchers encounter during this process. By the end of this presentation, participants will have a clear understanding of how to meet the Food and Drug Administration Amendments Act ( FDAAA 801 ) standards and ClinicalTrials.gov requirements, ensuring successful trial registration and reporting.  

Overview of the ClinicalTrials.gov database  

Which trials need registration & results reporting?  

How to Register a study, Report Results, and Update Records  

Common errors during registration, and results reporting, and how to avoid them  

Navigating the PRS Support System and Help content  

Overview of ClinicalTrials.gov PRS Beta website

Wednesday, September 18, 2024  

12:00 PM-1:00 PM  

Pre-register to receive the Zoom link  

IRB Efficiency Initiative Monthly Information Session

The Human Subjects Office invites the UI research community to attend monthly information sessions about the IRB Efficiency Initiative on the fourth Wednesday of the month from 12-1 pm, via Zoom. We will discuss the changes and demonstrate HawkIRB enhancements being implemented to streamline the IRB review process. The monthly sessions will cover upcoming enhancements and expected roll-out dates.   

Wednesday, September 25, 2024  

Office Hours  

Human Subjects Office staff host IRB Office Hours via Zoom to provide assistance with electronic IRB applications (in HawkIRB) and to discuss study proposals prior to submission.  All researchers are welcome to attend. No appointment is necessary.  

Fall Office hours will begin on September 4 th through December 19 th  

Wednesdays (2:00-4:00 PM) via Zoom    

Thursdays (10:00 AM-12:00 PM) via Zoom  

Recorded Training  

The IRB ICON Course for Researchers HawkIRB training sessions provide an orientation to the electronic IRB application and review system. These sessions are for anyone preparing to submit a HawkIRB application for the first time and for those who would like guidance about the proper completion of HawkIRB forms. Five recorded trainings are available 24/7.   

The recorded trainings are available on ICON at IRB ICON Course for Researchers .   

Note: HawkIRB training Parts 1 and 2 satisfy the Student PI Training Requirement.  

This paper is in the following e-collection/theme issue:

Published on 30.8.2024 in Vol 26 (2024)

Assessing the Feasibility and Acceptability of Smart Speakers in Behavioral Intervention Research With Older Adults: Mixed Methods Study

Authors of this article:

Original Paper

• Kelly Quinn 1 , MBA, MSLIS, PhD   ; 
• Sarah Leiser Ransom 1 , MA   ; 
• Carrie O'Connell 1 , MA   ; 
• Naoko Muramatsu 2 , PhD   ; 
• David X Marquez 3 , PhD   ; 
• Jessie Chin 4 , PhD  

1 Department of Communication, University of Illinois Chicago, Chicago, IL, United States

2 Division of Community Health Sciences, School of Public Health, University of Illinois Chicago, Chicago, IL, United States

3 Department of Kinesiology and Nutrition, University of Illinois Chicago, Chicago, IL, United States

4 School of Information Sciences, University of Illinois Urbana-Champaign, Urbana, IL, United States

Corresponding Author:

Kelly Quinn, MBA, MSLIS, PhD

Department of Communication

University of Illinois Chicago

1007 W Harrison St

1140 BSB, MC 132

Chicago, IL, 60607

United States

Phone: 1 312 996 3187

Email: [email protected]

Background: Smart speakers, such as Amazon’s Echo and Google’s Nest Home, combine natural language processing with a conversational interface to carry out everyday tasks, like playing music and finding information. Easy to use, they are embraced by older adults, including those with limited physical function, vision, or computer literacy. While smart speakers are increasingly used for research purposes (eg, implementing interventions and automatically recording selected research data), information on the advantages and disadvantages of using these devices for studies related to health promotion programs is limited.

Objective: This study evaluates the feasibility and acceptability of using smart speakers to deliver a physical activity (PA) program designed to help older adults enhance their physical well-being.

Methods: Community-dwelling older adults (n=18) were asked to use a custom smart speaker app to participate in an evidence-based, low-impact PA program for 10 weeks. Collected data, including measures of technology acceptance, interviews, field notes, and device logs, were analyzed using a concurrent mixed analysis approach. Technology acceptance measures were evaluated using time series ANOVAs to examine acceptability, appropriateness, feasibility, and intention to adopt smart speaker technology. Device logs provided evidence of interaction with and adoption of the device and the intervention. Interviews and field notes were thematically coded to triangulate the quantitative measures and further expand on factors relating to intervention fidelity.

Results: Smart speakers were found to be acceptable for administering a PA program, as participants reported that the devices were highly usable (mean 5.02, SE 0.38) and had strong intentions to continue their use (mean 5.90, SE 0.39). Factors such as the voice-user interface and engagement with the device on everyday tasks were identified as meaningful to acceptability. The feasibility of the devices for research activity, however, was mixed. Despite the participants rating the smart speakers as easy to use (mean 5.55, SE 1.16), functional and technical factors, such as Wi-Fi connectivity and appropriate command phrasing, required the provision of additional support resources to participants and potentially impaired intervention fidelity.

Conclusions: Smart speakers present an acceptable and appropriate behavioral intervention technology for PA programs directed at older adults but entail additional requirements for resource planning, technical support, and troubleshooting to ensure their feasibility for the research context and for fidelity of the intervention.

Introduction

The use of behavioral intervention technologies (BITs) in research has proven to be feasible and efficacious in a wide variety of settings [ 1 ], extending the range of research into new geographies and populations that were previously difficult to reach and by providing new media with which to develop and deliver interventions and record data [ 2 ]. Advances in artificial intelligence and computational linguistics have created a new class of technologies that can be used for these purposes. Powered by artificial intelligence and made accessible through voice user interfaces (VUIs), smart speakers or voice-activated personal assistants, such as Amazon’s Alexa and Google’s Nest Home, are widely available and readily acceptable to older adults [ 3 ]. Because of their utility and features, smart speaker technologies have become a focal point in gerontological and health research [ 4 , 5 ]. While attention has been placed on the use of websites, software, mobile apps, and sensors as intervention delivery mechanisms [ 6 ], less attention has been placed on the use of smart speakers as a BIT, especially for older adult populations [ 5 ].

Hermes et al [ 7 ] argue that BITs hold unique characteristics that should be evaluated distinctly as part of traditional implementation outcomes, with an emphasis placed on the evaluation of BIT at the consumer or participant level for factors such as acceptability and adoption. We further argue that BITs often comprise both a delivery technology, such as a smart speaker, and an underlying software application, which is the intervention itself. By conducting independent evaluations of intervention hardware and delivery technology and intervention software applications, a more accurate evaluation of implementation outcomes can be made.

This article aims to report on the acceptability and feasibility of using smart speakers to deliver an in-home physical activity (PA) intervention among a sample of older adults aged ≥65 years. Using data collected from surveys and interviews, along with researcher field notes and device logs, we focus on the evaluation of the smart speaker device, and not the intervention application, as a BIT delivery mechanism in a 10-week pilot study that used Google Nest Home Mini smart speakers. The contribution of this study is 2-fold. First, it examines the feasibility and acceptability of smart speakers as an emerging component of BIT delivery systems, independently of an intervention assessment. Second, it examines the appropriateness of smart speaker technology for use in PA interventions for older adults.

Smart Speaker Basics

Smart speakers use a VUI to aid users in navigating everyday tasks, such as finding information, scheduling events, setting timers and alarms, and playing media [ 8 , 9 ]. The VUI language processing system, also called a voice assistant or conversational agent, is the defining characteristic of the smart speaker. To accomplish tasks by voice, smart speakers integrate several different technologies into a single device to leverage dialogue capabilities: these include subsystems for voice recognition, natural language processing (understanding and generation), and cloud-based data processing. Typically activated using a wake word or phrase, such as “Hey Google” or “Alexa,” smart speakers remain in a state of ambient listening or are always “on,” waiting for the users to initiate a conversation or command. Energy-efficient processors passively process, or “listen,” for the wake word, buffering and rerecording within the device without transmitting or storing any information [ 10 ].

As illustrated in Figure 1 , once a wake word is detected, the device is triggered to begin actively recording [ 11 ], transmitting recorded requests to the device maker’s cloud-based service to decipher users’ speech [ 12 ]. Cloud-based data processing and storage alleviates the need for the device to be capable of speech recognition [ 11 ] or file storage [ 13 ], and data are transmitted seamlessly between the device and the cloud. The audio transmission is deciphered into commands using a natural language processing algorithm, and an appropriate response is generated using speech synthesis, then sent back to the smart speaker to be conveyed to the user [ 12 , 14 ].

To enhance device utility, most platforms like Google and Amazon encourage personalization of the activities that can be performed on their devices. Users are urged to create user and voice profiles and to share personal information like home and work addresses, credit card numbers, calendars, account logins, transportation modes, and nicknames. This information is then used to streamline activities that can be facilitated through the device, such as purchasing items, setting calendar reminders, and generating shopping lists. In another form of customization, Google and Amazon provide developers the ability to build and market add-on applications called skills (for Amazon’s Alexa) and actions (for Google Assistant), which augment available native applications. These custom actions can be designed to support research activities by recording data, supporting intervention activities, or reminding participants to pursue specific actions.

Older Adults and Technology Acceptance

Technology has become increasingly important to everyday life, yet older adults can often trail behind in the adoption of new technology because of limited experience and a lack of necessary skills [ 15 , 16 ]. While age-related gaps in internet use narrowed significantly since the pandemic, older adults still lag behind other age groups in both use and access to broadband connections, as compared with those in younger age groups [ 17 , 18 ]. In part, this may be due to experience with technology in the workplace [ 19 ], but declines in physical and cognitive abilities and limitations in the performance of instrumental activities of daily living may also lead to decreased use [ 20 , 21 ]. Although the use of technology by older persons can often enhance perceptions of life quality [ 22 , 23 ], socioeconomic factors, such as lower income and education levels, compound age-related differences [ 24 ].

VUIs represent a class of technologies that are readily accepted by older adults and perceived as easier to learn and use than keyboard interfaces [ 3 , 25 , 26 ]. Because VUIs do not rely on vision or touch, they are accessible to those with visual or fine motor degradation, mobility impairment, and disability [ 27 ]. In addition, older adult users often build companionship with VUIs, which results in positive experiences that may not only lead to reduced loneliness and increased independence [ 28 ] but also may help to overcome frustration with technological errors [ 29 ]. Processes of technology acceptance by older adults often reflect the dynamics of technology adoption and use described by the Technology Acceptance Model and its derivatives [ 30 ], including the widely accepted Unified Theory of the Acceptance and Use of Technology (UTAUT) model [ 31 ].

Older Adults and PA

The health benefits of PA for older adults are well documented [ 32 ], and its importance in supporting healthy aging cannot be overstated. PA slows age-related declines in functional abilities and helps to maintain physical and mental capacities in such diverse areas as muscle strength, cognitive functioning, disease prevention, and anxiety and depression reduction [ 33 ]. Despite these benefits, older adults report high levels of sedentary behavior [ 34 ]. Environmental contexts, such as weather, accessibility because of distance, and cost and affordability, are cited as factors by older adults for not being physically active [ 35 ]. It is often challenging for older adults to adopt and adhere to a PA regimen [ 36 , 37 ], and mobility challenges sometimes limit the ability to regularly participate in community-based programs because of accessibility issues [ 38 ].

Behavioral interventions to increase PA among older persons are largely successful in increasing levels of PA, and studies have shown that older individuals are more likely to continue PA programs in home-based settings [ 39 , 40 ]. Internet-based PA programs are both cost-efficient [ 41 ] and effective in producing behavioral change [ 42 ], and home-based programs have better adherence rates than community-based programs [ 40 ]. Technology-based interventions have been effective in producing a change in PA behaviors when compared with traditional mechanisms, such as usual care, minimal contact, waitlist control groups, in-person, or other nontechnology interventions [ 43 ]. Taken together, these factors suggest that a home-based PA program facilitated by technology, such as a smart speaker, could support the PA readiness of sedentary older adults. This study contributes to the extant literature by reporting on the acceptability and feasibility of smart speakers to deliver PA programming among a sedentary group of older adults.

Smart Speakers as a BIT

Smart speakers are emerging as a locus in behavioral intervention delivery systems [ 5 ]. Smart speakers can be deployed in participants’ local environments and enable interventions to be delivered remotely, thereby reducing barriers to administration and adherence. When evaluating the use of technology in implementation research, Hermes et al [ 7 ] argue that criteria used to evaluate intervention implementations [ 44 ] should be applied to BITs and the strategies used to guide their use apart from processes used to evaluate intervention implementation, as by conducted these separately a more accurate evaluation of implementation outcomes can be made.

It is relevant to note that the Hermes et al [ 7 ] approach ignores potential distinctions between an intervention mechanism, which may be a technology, such as a software program or application through which the intervention is delivered, and its underlying hardware and delivery infrastructure, which may include a device, such as an internet-enabled watch or smart speaker through which the intervention software operates, as well as the Wi-Fi or internet signal on which it is dependent. We argue that distinctions between each of these elements are important to make when evaluating the feasibility and acceptability of a BIT as well, as challenges and successes may occur with any of the components. Moreover, it is relevant to consider a participant’s ability to distinguish between these elements when undertaking this analysis to ensure appropriate identification of evaluation criteria. For example, studies on smart speakers used for general purposes have suggested that some users fail to make distinctions between the various components of service delivery, such as the voice interface and the device [ 45 , 46 ], which leads to conflation between perceptions of each element.

Five criteria in particular are recommended for evaluation from the perspectives of the participant or consumer and providers or researchers [ 7 , 44 ]:

• Acceptability, or the extent to which a technology is useful or satisfactory.
• Adoption, or the intention to use the technology.
• Appropriateness, or perception that the technology fits, is relevant or compatible with the context of its use.
• Feasibility, or the extent to which a technology can be successfully used in a specific context.
• Fidelity, or evidence that technology can be delivered as intended.

When evaluating a BIT at the level of the consumer or research participant, the outcomes of acceptability, feasibility, and adoption are most commonly measured through models of technology adoption [ 7 ]. The UTAUT model [ 31 ] is one of the most widely used theories of technology adoption [ 47 ], connecting the concepts of acceptability and feasibility to a third important criterion from the user’s perspective: adoption. UTAUT argues that a causal relationship exists between users’ perceptions of technology and their intention to use it. It specifically identifies 4 constructs—expectations of a technology’s performance, the ease with which it can be used, social influence, and facilitating conditions—and links these to a user’s intention to use a particular technology, which, in turn, is strongly correlated with its use. UTAUT subscales regarding performance and effort expectancy have been proven reliable with respect to the use of a wide variety of technologies in older adult populations, including email and social media [ 48 ], tablet computers [ 49 ], and remote health care or telehealth applications [ 50 ].

Studies on older adults’ use of technology have found that expectations of a technology’s performance and perceptions of the amount of effort that will be required to use technology are powerful incentives for new technology adoption [ 51 , 52 ]. Although the feasibility of using smart speakers among older adult populations to improve well-being has been examined in several recent studies [ 25 , 28 , 53 ], their use to explore specific interventions is more limited, especially among older adult populations.

Study Design

Participants were engaged in a 10-week, evidence-based, internet-based PA program that used artificial intelligence to guide activities from October 2020 through January 2022. Of note, the time frame of the study coincided with the COVID-19 pandemic and consequent lockdowns. Because all in-person leisure activities had ceased, the use of alternative delivery mechanisms for exercise programming, such as smart speakers, was potentially attractive. Though the study had been initially designed and planned to include an in-person orientation to the smart speaker and PA application, the pandemic necessitated that all participant interactions be carried out in a virtual context because of the particularly vulnerable nature of the target population. Consequently, all participant interactions, including onboarding and offboarding, weekly check-ins, and interviews, were conducted remotely via Zoom when possible and alternatively via phone calls.

A PA application was developed by the research team to run on Google Nest Home Mini speakers, which replicated components of the Healthy Moves for Aging Well program [ 54 ]. A description of the activities included in the PA application can be found in Multimedia Appendix 1 . Initial feasibility and acceptance of the PA application were validated in a pilot user study before field deployment [ 39 ].

To ensure that participants had internet access, we distributed Wi-Fi hot spots together with the smart speakers. The hot spots also enabled the research team to perform the initial speaker setup and installation of the PA application before the first session. This simplified the orientation process for participants, as the devices merely needed to be connected to power for participant use.

Recruitment was aided by a partner senior-living organization located in a large suburban Midwestern county, which disseminated recruitment materials to both their independent living facility residents and via community programming information channels. The research team consisted of 4 individuals (1 principal investigator and 3 graduate research assistants, all adult women), certified by the Collaborative Institutional Training Initiative Program [ 55 ]. The study design is illustrated in Figure 2 .

After enrollment, participants were sent a parcel containing written instructions regarding the use of the smart speaker, PA program application, and Wi-Fi hotspot; a smart speaker; a Wi-Fi hot spot for in-home internet connection diary materials; and consent documents. Initial participant meetings were conducted by phone or Zoom, during which baseline PA and technology attitude measures were taken, and verbal instructions were shared on using the smart speaker, PA app, and Wi-Fi hotspot. Participants were randomly assigned to a research team member who assisted with equipment setup and conducted all interviews and weekly check-ins throughout the intervention. Functionally, this meant that each researcher worked with the same 5 to 6 individuals throughout the study. At baseline (T0), participants took part in a brief individual motivational coaching session to determine their PA goals.

The intervention then took place in 2 phases. During phase 1, which lasted 6 weeks, participants were encouraged to use the smart speakers for their purposes (eg, answering questions, playing media, setting timers or alarms), as well as use the PA application for a minimum of 3 sessions per week. Participants were contacted weekly by phone by a trained member of the research team during this phase to troubleshoot, assess PA goal achievement, and set new PA goals for the coming week. Phase 2 lasted 4 weeks, during which participants continued natural use of the smart speaker and the PA application on their own (ie, without weekly contact).

Assessments of the participants’ perceptions of technology acceptability, prior technology experience, and attitudes toward technology were administered by the researchers at 3 points in the study: T0, 6 weeks and end of phase 1 (T1), and 10 weeks and end of phase 2 (T2). Semistructured interviews provided perceptions and use of the smart speakers and the PA application at T1 and T2. Device logs were maintained and reviewed for the entire study period. Participants were compensated after each interview (T0, T1, and T2) and received a completion bonus for completing all visits.

Ethical Considerations

The study protocol was approved by the institutional review board of the University of Illinois Chicago (UIC Protocol 2019-1013), which reviews and approves human subjects research in accordance with the ethical principles outlined in the Belmont Report and the DHHS regulations 45 CFR Part 46. All procedures were carried out as specified in the study protocol. All participants provided oral acknowledgment of informed consent, as a written acknowledgment requirement was waived because of the conditions of the pandemic. All data were deidentified before analysis. Participants were compensated up to US $100 for time spent and were able to keep their smart speaker device on the study’s conclusion. The reporting of the qualitative findings follows the guidelines outlined in the Consolidated Criteria for Reporting Qualitative Research [ 56 ] for reporting health-related studies, as appropriate.

Data Sources

Measures of technology acceptability and feasibility.

Researchers administered a questionnaire with quantitative measures of familiarity with technology, UTAUT, perceived sociability, and social presence during the participant interviews at T0, T1, and T2. The UTAUT subscales consisted of 30 items related to performance expectancy, effort expectancy, attitudes toward technology, social influence, facilitating conditions, smart speaker self-efficacy, anxiety, and behavioral intention to use smart speakers, measured on a 7-point scale and adapted to the context of smart speakers [ 31 , 39 ]. Four items were used to measure the perceived sociability of smart speakers, and 5 items assessed the social presence of smart speakers [ 57 ]. Items for each measure are listed in Multimedia Appendix 2 . To indicate technological competence in everyday living, a familiarity with technology measure was adapted from the Everyday Technology Use Questionnaire [ 58 ] regarding the frequency of use of 11 everyday technologies, such as searching the internet for information, dealing with recorded telephone menus, or sending and receiving emails [ 59 ].

Device log data were collected from the Google accounts attached to the devices; however, data from 2 (11%) of 18 participants was missing because of technical errors. The remaining (16/18, 89%) data logs were analyzed for frequency of engagement with the PA app by calculating the number of times “Healthy Moves” was activated by the participant. To avoid fatigue, the PA program application asked participants if they needed to take a break or stop the exercise activity. Consequently, because participants were able to voluntarily truncate the application, we used its activation as a measure of engagement.

Interviews and Field Notes

A total of 36 interviews were transcribed for analysis, with an average duration of 18:05 minutes for interviews at T1 and 17:06 minutes for interviews at T2. The interview guide is presented as Multimedia Appendix 3 . In addition, 90 sets of field observation notes were recorded after the weekly motivational sessions during phase 1 and examined.

Because recruitment took place during pandemic lockdowns , participants were recruited using social media postings, recruitment emails posted on listservs, and through the use of 2 research recruitment matchmaking portals. Inclusion criteria were those aged ≥65 years and who spoke English, as the smart speaker application was programmed in English. Participants were excluded if they participated in ≥150 minutes of PA per week or scored <7 correct responses on the Short Portable Mental Status Questionnaire [ 60 , 61 ].

A total of 24 participants were enrolled at T0. The size of the sample was largely determined by the availability of respondents and the adequacy of resources to complete the study. Because of interest in maintaining a PA regime and health complications, 4 participants did not complete the T1 interview. An additional 2 participants did not complete the T2 interview because of withdrawal from the study, so the final sample consisted of 18 individuals. The final study completion sample had an average age of 75.9 (SD 7.3) years; 15 (83%) of 18 participants were female, and 17 (94%) of 18 were White. Familiarity with technology was high among these participants, with 14 (78%) of 18 using a range of everyday technologies, such as searching the internet, using email, or sending text messages on a mobile device, on average, more than once per month. Table 1 summarizes the characteristics of both the study completion sample (n=18) and the recruited sample (n=24).

a Adapted from Everyday Technology Use Questionnaire [ 58 , 59 ].

Data were analyzed using concurrent mixed analysis [ 62 ], with log data, interview, and field note data examined for complementarity and completeness [ 63 ] to broaden and enrich the understanding of the quantitative measures of device feasibility and acceptability. Because of this choice of method, only data collected from the study completion sample were used in this analysis. SPSS (version 29) was used for repeated measures ANOVA, and MaxQDA 2022 (version 22.08.0; VERBI GmbH) was used for first-level descriptive and second-level thematic qualitative text analysis of interview and field observation data.

The coding schema for qualitative analysis of the interview and field note date was developed using a hybrid approach, with a priori codes from the UTAUT constructs comprising the initial coding structure and additional codes developed in vivo. First, a descriptive analysis of the texts was performed by a team of 2 researchers, coding approximately 9 transcripts each. Salient words and phrases were highlighted, and special attention was paid to adjectives and adverbs that added emphasis or provided a judgment of value. Next, keywords and phrases were grouped under thematic umbrellas that either aligned with the preestablished categories derived from UTAUT codes or newly identified categories that emerged during the first descriptive coding phase. The collected measures were then grouped and analyzed according to the criteria for assessing BIT [ 7 , 44 ] to provide an overall assessment of acceptability, adoption, appropriateness, and feasibility. Acceptability was assessed using the UTAUT subscale of performance expectancy, along with measures of perceived sociability, pleasantness, and social presence. Adoption was measured using device log data on intervention app use and examination of the subscale on the behavioral intention to use the smart speaker. Appropriateness was evaluated using measures of attitudes toward using smart speakers and smart speaker self-efficacy. Feasibility was appraised using the UTAUT subscale of effort expectancy.

As the data analyses were conducted concurrently using mixed methods, the results are also presented concurrently. In this format, log data, interview data, and field observation data offer complementarity and completeness to the quantitative measures of acceptability and feasibility.

Acceptability

Adults in this study perceived the smart speaker as acceptable, with measures of performance expectancy (ie, perceived usefulness) to be high during the study period (T0: mean 5.81, SE 0.16; T1: mean 5.04, SE 0.31; T2: mean 5.02, SE 0.38). However, perceived usefulness decreased over time ( F 2,34 =3.66; P =.04; partial eta squared=0.18), perhaps reflecting the practicality of actual use once participants integrated the smart speakers into their everyday routines.

Participants expanded on these perceptions in the interviews, noting that specific features of the smart speaker encouraged these perceptions of utility and spurred them to engage in more frequent and routine use of the PA program. One factor they noted was that because the smart speaker was located at home, barriers to PA engagement were reduced:

Convenience. I didn’t have to go out of the home for the exercise. [Participant 8]

Its accessibility and yeah, I guess that would be it. Its accessibility. It’s right there and it’s easy to consult and makes it easier to engage in the [PA] program. [Participant 19]

The smart speaker also afforded participants time flexibility, enabling them to engage in the intervention at times that were convenient or available for them instead of a set or designated time. This time flexibility enabled them to adhere to the protocol with greater success:

I tried unsuccessfully to pick a time of day that worked the best and stick with it... In a way, it was an advantage because I could fit it in whenever it occurred to me or that I was reminded in some way. [Participant 5]

The visual and physical presence of the smart speaker device also encouraged intervention adherence for participants, as it “reminded” participants to engage in the intervention activities, which also improved adherence:

I like the fact that it reminds me, just seeing it reminds me to do it, and I don’t know what I would use in its stead. [Participant 5]

Several participants noted that the VUI interface of the smart speaker added to the device’s convenience by making it expedient to use:

I don’t have to type anything in, I can just talk to it and it gives me the quick answer that I’m looking for… I think it’s faster than the computer. Because it’s quicker to talk than for me to type it. [Participant 9]

Participants perceived the smart as being pleasant to interact with throughout the study period (T0: mean 5.89, SE 0.16; T1: mean 5.27, SE 0.31; T2: mean, 5.03, SE 0.38), but this perception decreased over time ( F 2,34 =3.63; P =.04; partial eta squared=0.18), perhaps because the novelty of interaction diminished over time. As one participant noted:

So I think it’s a novel way of getting one to focus on an exercise program like this, and to be able to look, shall we say, look forward to doing it, more than if it were simply something you were reading from a pamphlet. [Participant 13]

They also perceived low social presence of the device (T0: mean 3.82, SE 0.27; T1: mean 3.52, SE 0.34; T2: mean 3.31, SE 0.35), and the low social presence did not change over time ( F 2,34 =1.27; P =.29; partial eta squared=0.07).

Adoption of the smart speakers, or the intention to use them, was evidenced through actual use of the devices and indication by participants of a willingness to use its functionality for other purposes in addition to the PA intervention. Examination of the device log data revealed that while heterogeneity existed in the PA program engagement, the use of the devices decreased over the study period. For the first 6 weeks (phase 1), participants engaged with the intervention app <2 times per week (mean 10.19, SD 13.26; range 0-42). Activity was higher during this initial period, perhaps because of the accountability provided by the weekly check-in calls from the research team. Engagement with the intervention app dropped off significantly during the second phase of the study, with participants engaging with the intervention app approximately about once per week (mean 4.94, SD 7.85, range 0-29; F 1,15 =9.49; P =.008; partial eta squared=0.39). However, the patterns of use were heterogeneous, with some participants engaging with the device frequently and others only minimally.

About half of the participants engaged with the PA intervention less than one time per week during the intervention and follow-up phases (n=9), but other participants (n=4) were quite active, engaging with the intervention >4 times per week during phase 1, and continued engagement with intervention >2 times per week during the phase 2. The most frequent user engaged with the PA intervention every day during phase 1 and even more than once daily during phase 2. Examination of the interview data reinforced these findings and demonstrated how these 2 clusters differed. It also reinforced the understanding that as expectations of the device met user experiences, regular use of the devices was reinforced. As one participant noted:

Well, at first I just used it for the exercise program, and then I was a little bit more daring and listened to some music, and then jokes, and the weather, and timer and things like that. I became more comfortable using it more often. [Participant 1]

Conversely, when participants did not use the device with any regularity for either exercise or everyday activities, they were less likely to indicate that they would continue using it at all. The lack of engagement with the smart speaker appeared to lead to a lack of adherence to the intervention protocol:

[B]ut I’m a little distressed with myself for not even thinking of it this week, yeah. Well it didn’t seem to be too helpful to me when I was using it, I guess that’s why I forgot about this week. [Participant 2]

Participants demonstrated a strong willingness to continue using the device during and after the study period (T0: mean 6.35, SE 0.11; T1: mean 5.98, SE 0.27; T2: mean 5.90, SE 0.32), and this behavioral intention to use the device did not change over time ( F 2,34 =1.48; P =.24; partial eta squared=0.08). We note that participants who made a concentrated effort to incorporate the device into their daily routine during the study, for both exercise and other activities, demonstrated greater intention to continue using the device, even beyond the scope of the study.

Appropriateness

Appropriateness, or the perception that the technology is compatible with the context of its use, was demonstrated consistently during the study period, though these perceptions were not uniform among participants. Participants held strongly positive attitudes toward smart speakers (T0: mean 6.17, SE 0.09; T1: mean 6.07, SE 0.16; T2: mean 6.00, SE 0.18), and these perceptions were sustained throughout the study ( F 2,34 =.82; P =.45; partial eta squared=0.05). Comments about specific qualities of the smart speakers that enhanced the delivery of the PA intervention elaborated on these positive perceptions. For example, one prominent feature that was frequently mentioned by participants was the VUI. Participants found the auditory instructions provided by the smart speaker to be an appropriate mechanism to deliver the PA intervention, likening it to an individual coach or mentor:

Well, I do like the fact that even though this is not a real person, there is a voice telling you what to do. So you’re not in a gym filled with people or you’re not in a class. You don’t have to go anywhere, but someone is standing there telling you what to do or sitting there telling you what to do. And so that is a, I think for persons who say live by themself or something, it is like another voice that, I guess it’s watching a TV too, but it’s a voice that coaches you on. [Participant 24]

Other participants liked the ability just to listen and follow instructions. The simplicity of following commands enabled them to carry out the activities easily and with minimal cognitive effort:

I don’t have to think about it because I just follow whatever the directions are. When we’re doing the other exercise, I have to use the paper and I have to look at the paper. In here, you just follow the directions and you’re all set. [Participant 7]

However, the VUI may also have presented some hindrances for delivering the PA intervention. Prior work has identified that older adults may experience difficulty with constructing a structured sentence command in smart speaker use [ 29 ]. Participants in this study described how they had to “learn to talk to the device” by rephrasing questions and commands to obtain a desired response:

But every now and then on the [PA application], I think you’re supposed to answer a certain way. If he [the smart speaker voice] says, “Are you ready to go to the next exercise?” Then you go “Uh-huh.” And he’s like, “Did you hear me?” I think maybe you should know the respect that he needs because sometimes I think he doesn’t understand me and I forget to speak clearly. [Participant 11]

Participants consistently perceived that they could effectively use the smart speakers, assessing their self-efficacy in using the smart speakers quite positively (T0: mean 6.03, SE 0.12; T1: mean 5.79, SE 0.26; T2: mean 5.58, SE 0.25). However, this self-efficacy assessment dropped during the study period ( F 2,34 =1.81; P =.18; partial eta squared=0.10) as participants integrated the devices into daily routines. Self-efficacy was also demonstrated by participants’ ability to engage in intuitive workarounds for issues of communication with the smart speakers. These workarounds consisted of repeating or rephrasing commands, adjusting the speaking volume, or articulating the command more clearly:

...I don’t know whether it was the way I asked, ya know, I asked the question and the device said, “I didn’t understand the question.” So I said it a different way and then it was able to answer. [Participant 7]

Well, once in a while, I think maybe I’m not close enough to it and they will say, “I can’t understand you.” And I don’t know whose fault that is, if it’s a Mini Google device or maybe I’m not directly in front of it. But then I try again and I’m able to interact with it. [Participant 12]

However, challenges with the ability to engage with the intervention on the smart speaker may have contributed to participants’ feelings of being less proficient in using the smart speaker as the study progressed. Miscommunication issues with the device appear to be the primary hindrance and sometimes result in the participant ceasing interaction with the smart speaker or taking time away from the device before attempting interaction again. This effectively prevented the participant from carrying out the intervention at the desired or appropriate time:

Sometimes the Google device misses the mark as far as giving me exactly what I’m looking for. So I either have to rephrase what I’m asking or give up. [Participant 13]

Other participants, especially those who used the device irregularly or only for the PA application, were often discouraged when faced with these difficulties. Some participants did not seem comfortable adjusting their language or speaking style to accommodate the device and would disengage from the conversation and walk away:

[B]ut when I tried to get onto the [PA application], I tried it twice and then both times it said it wasn’t responding, but I didn’t get frustrated. I just thought, “I’m not going to do exercises today.” [Participant 1]

Feasibility

Feasibility, or the perception that smart speakers are easy to use, remained high during the study period . Participants reported that they found the smart speakers easy to use (T0: mean 5.93, SE 0.53; T1: mean 5.56, SE 1.16; T2: mean 5.55, SE 1.24), and this ease of use was sustained throughout the study ( F 2,34 =1.49; P =.24; partial eta squared=0.08). As one user summarized in the interviews:

I found it very easy to use. Very, very helpful. It got me to exercise three times a week, more than I could have without it. [Participant 1]

From the perspective of the research team, however, feasibility was not as clear cut. Examination of field note observations revealed that during the weekly contacts in phase 1, members of the research team often assisted participants in effectively using their smart speakers. Though participants reported that their devices were easy to use, researchers noted that they had made suggestions that devices be moved within the participant’s residence to improve the Wi-Fi signal (n=5) and that participants might rephrase commands to the device to gain an appropriate response (n=5), or worked with participants to adjust volume settings (n=2). In addition, researchers suggested and encouraged participants to use the device in alternate ways in addition to the PA program app (n=4), such as seeking information about the weather or setting timers and alarms. The frequency and nature of these observations suggest that providing support for using the devices was a necessary element in the research protocol and required resources from an administrative perspective, including training members of the research team in basic device functionality as well as potential troubleshooting strategies.

Wi-Fi connectivity issues were a major determinant of the problems that participants experienced. When smart speakers have difficulty maintaining a continuous internet connection, especially when deployed over Wi-Fi hotspots, they may encounter limits on bandwidth, which can impair their operation [ 64 ]. The use of Wi-Fi hotspots was a source of some of the connectivity issues experienced by participants and caused frustration in the form of longer-than-anticipated response times or interruptions in the performance of the device. These signals of connectivity issues required patience when experienced:

I would be patient. Sometimes there’s a long pause before it responds. So patience is sometimes necessary, or just repeat my request. And if I get a sense that it’s not being digested by the device, I rephrase it. [Participant 13]

The placement of the smart speaker within the participant’s residence was also a critical factor in ensuring successful interactions with the device. Prior research has identified that older adults strategically position smart speakers to shape and organize daily routines [ 65 ]. Participants frequently placed devices in the area of their residence, which they spent the most time in to optimize accessibility and convenience. Often, this meant that the smart speaker was placed in a living or family room, but the kitchen was also a popular option. However, these spaces were not always optimal for participating in the PA program intervention. As one participant noted:

Now, I’m sitting in a computer room a study. And so, a lot of times I do it [the PA program application] in here. In fact, I think I’ve been doing it in here for the last week or two, but sometimes I put it [the smart speaker] in the area where we eat, the kitchen eating area, so that if I’m working in the kitchen and I’d like to use it for a timer. So, then I, because it’s there, I ended up doing the exercises in the kitchen. [Participant 16]

Privacy considerations also play a role in device location and, again, may run counter to optimal placement for engagement with the intervention activity or a strong Wi-Fi signal. As one participant described:

My husband and I worry about the privacy issues with the Google Home Mini, and several times when I had it plugged in to prepare in order to do exercises, I would mention, “Oh, I have to Google that.” And the Google Mini would come on. And that was very disconcerting because I was in another room and it was listening. [Participant 21]

Ultimately, participants seemed to maximize the opportunity to use the smart speakers and tried to place the device in ways that would facilitate their engagement with the intervention:

I kept it right here in the living room where it was visible to me every day. If I had tucked it away somewhere, that would have been even more problematic on my part because it’s easy to forget. [Participant 5]

Fidelity, or evidence that the technology can deliver the intervention as intended, is a criterion that is perhaps best viewed from the perspective of the research team, but participants noted how the smart speaker enabled them to execute the intervention with greater precision. Participants liked the program’s routine and noted that it provided structure, including PA, in their daily routines.

I like the fact that it times you, in other words, I don’t have to keep track of the time with the clock or anything like that. I like, it just gives a little more structure so that I can rely less upon my own motivation. I would say it serves as a motivator too. [Participant 9]

In addition, some participants noted that the smart speaker introduced accountability into their engagement with the intervention, alluding to its potential to be used as a mechanism to report on activities associated with the study:

And I realized that I’m on my own time. I can do this or I don’t have to do it, but having the device makes me feel responsible that it’s kind of like big brother is watching me, and if I don’t use it, it’s going to know. [Participant 18]

Though the PA application had been tested extensively before deployment in the study, issues with its performance and interoperation with the device were encountered during both phases of the study. Participants reported that they experienced technical difficulties with the PA application, including the application not responding to them in a timely manner, aborting the intervention without warning, or repeating exercises that they had already completed. Minor modifications to the application by the development team during the study resulted in some improvements in performance, but most incidents were attributable to Wi-Fi connectivity issues, which could cause the PA program application to cease functioning and impair the fidelity of the intervention.

Participants found incidents very frustrating and for some, induced nonuse of the intervention application. As one user stated:

I tried to do it with the (PA application), but it doesn’t work right for me, so I just don’t use it then. So, I do a little exercise, but I just don’t use the Google, I just do them on my own. [Participant 20]

As this participant suggests, when encountering technical issues with the PA application, participants often maintained the exercise regimen of the study without the assistance of the smart speaker. Participants had received extensive support materials in their orientation packet, including illustrated descriptions of the activities that were invoked by the PA application. Because of these materials, several participants felt that they could complete the exercises without the guidance of the smart speakers; however, some used the functionality of the device, such as a timer, to assist in carrying out their activities:

[Participant] told me when she gets frustrated with that she will just use the Google Home [smart speaker] to set a timer and go through the exercises on her own. [Field notes, Participant 16]

Participants also appeared to be able to distinguish between the PA application and the hardware of the smart speaker when they reported the challenges they encountered. This was made clear through their use of gendered pronouns when speaking about the various components, identifying the male voice of the PA application and the female voice of the smart speaker interface, as well as specific references to the PA intervention as an “app.”

Well, there were problems with the software where it would abort and I would be shifted over to the regular Google application as opposed to the [name of PA program app]. [Participant 13]

Taken together, these last 2 points suggest that when participants’ challenges with the PA application were technical, they recognized it was a limitation of the intervention delivery mechanism and not the smart speaker or the intervention. Instead, they sometimes resorted to workarounds, such as relying on their memory to complete the exercise or using the support materials as a prompt. This underscores the importance of providing support materials to participants, not only for the delivery mechanism but also for the intervention itself.

Principal Findings

The goal of this paper was to evaluate the use of smart speakers to deliver an in-home PA intervention among a sample of older adults using previously established criteria for the evaluation of BITs. Our focus for this analysis was on the smart speaker as a delivery mechanism and not the PA intervention itself, as these are distinct components of the intervention implementation and should be evaluated separately. The smart speakers in this study were found to be highly rated for a PA program by participants regarding acceptability, appropriateness, and feasibility, criteria essential to quality BITs [ 7 , 44 ]. Functional and technical factors related to the operation of the smart speakers, such as ensuring consistent Wi-Fi connectivity and ensuring participants used appropriate phrasing when interacting with the devices, created a responsibility for the research team to provide basic technical support and troubleshooting resources. In addition, these same factors possess the potential to impair the fidelity of the intervention. In short, while smart speakers provide a novel and acceptable technology for intervention research, their feasibility in a research context comes with limitations.

Smart speakers afforded participants in this study convenience and flexibility for engaging with the intervention activities and served as a visual reminder to reinforce completion of the study protocol, which improved adherence to the intervention. The VUI was well-received by participants, who noted its ease of use and appropriateness for coaching participants through a PA program. The VUI also introduced challenges for the participants, as it required them to learn how to appropriately phrase commands and adjust their speaking volume to communicate with the device. This represents an intriguing intersection of possibility and limitation. On the one hand, the benefits of the device, as articulated by most respondents, are based on its ease of use and convenience, which are associated with being a hands-free interface that is capable of responding quickly and specifically. This presents a range of possibilities for intervention-based research across health, education, and other applications for older adults [ 66 , 67 ].

By contrast, at the current stage of development, the smart speakers used for this study are not capable of accommodating a human user’s natural diction and phrasing beyond stating “I’m sorry” and requesting the user rephrase their question or direction until an acceptable rudimentary keyword or phrase is recognized. Therefore, successful accommodation to miscommunication hinges on the ability of the participant to mold their habits and language to patterns recognizable by the device. When participants were flexible about adjusting their phrasing and behaviors to mitigate technical glitches, they were also more likely to view the device favorably and use it regularly. When considering acceptability, this is particularly meaningful because the limitations of the technology and the adaptiveness of the participant base must be evaluated in tandem. Future development in artificial intelligence–supported health interventions could leverage the advancement in large language models to provide ubiquitous and fluent user experience [ 68 ].

Participants indicated their intention to use a smart speaker through positive attitudes toward its functionality, but there was a high degree of heterogeneity in their adoption. Some participants embraced the use of the devices, whereas others were frustrated and abandoned their efforts easily. Issues related to Wi-Fi connectivity were particularly challenging and interfered with the ability of the device to function appropriately. The feasibility of using smart speakers in an intervention, while positive from the participants’ perspective, was more challenging from the vantage point of the research team. To provide technical and operational support for successful device operation, members of the research team were required to be familiar with device functionality and basic troubleshooting strategies. In addition, because participants had discretion over device placement within their residence, it was more challenging to ensure that the device would be optimized for both execution of the intervention and Wi-Fi connectivity. These connectivity issues impacted intervention fidelity and underscored the need for robust support.

Overall, technical issues, such as glitches with the PA application, device-level technical problems (volume, articulation, etc), and broader critical infrastructure issues, such as a weak Wi-Fi signal, will stymie engagement with intervention activity and broader intentions to use the smart speakers. However, when participants have outlets to seek technical assistance and support when issues arise, such issues can be effectively mitigated. In other words, older adults are not at all resistant to engaging with smart speakers; however, a robust technical and informational support system should be in place.

The privacy considerations for smart speakers are not inconsequential. To be activated, smart speakers typically require an account to be established with the device maker, which is then associated with the smart speaker. Often, these accounts require additional information to be gathered from the user, such as credit card details, causing concern about personal information collection [ 69 ]. Establishing a linkage between the account and the smart speaker enables personalization of the activities that can be performed, such as reminders; however, it also associates this same information with voice recordings and device interactions [ 13 ]. Location-based data, such as time zone and zip code, allow device makers to transmit relevant information, such as weather and traffic news, but these data also become associated with accounts. All of these additional data points may increase participant privacy vulnerability.

One strategy to enhance participant privacy is to use pseudonym accounts to set up the smart speaker and collect data, as was done in this study. This approach provides some ability to shield participants’ identity, but it also reduces the functionality of smart speakers considerably, as devices are unable to be personalized for calendaring and reminding functions. This reduction in utility can be frustrating to study participants, who anticipate a level of functionality frequently advertised by device makers. However, ultimately, privacy concerns are associated with smart speaker use [ 70 ], and those who use them may be qualitatively different than the general population [ 71 ], thereby presenting a form of sample selection bias for research using the devices.

Limitations

Data analyzed in this study were collected from a relatively small, not randomly controlled sample, so the conclusions reached may not be generalizable to a wider older adult population. This study was conducted during the pandemic, which required recruitment to be carried out via the web, and interviews were conducted via telephone or Zoom. The implications of this context are 2-fold. First, this sample may have been more accepting of technology and may have had higher educational attainment than if they had been enrolled in a clinical or face-to-face setting. This might suggest that the issues highlighted with this sample may be even more pronounced with populations with lower technological proficiency or education levels and may require researchers to provide additional support resources when carrying out studies of this nature. Second, the context of the pandemic may have prompted the participants to have greater acceptance of smart speaker technology to engage with PA, as in-person activities were significantly reduced during that time. These factors also limit the generalizability of these findings as they may have introduced a positive bias to perceptions of acceptability and feasibility in this sample. To gain a greater and more nuanced understanding of the acceptability and feasibility of using smart speakers in a research context, additional studies with representative samples in a nonpandemic context are required.

Challenges encountered by participants that were related to the wireless connectivity of the smart speakers may also have influenced perceptions of the smart speaker and the intervention application. Further exploratory work is needed to distinguish how perceptions of applications can be distinguished from their related delivery mechanisms for evaluation of usability, feasibility, and efficacy. In addition, data collection was constrained because of the pandemic, which limited the ability to observe participants engaging with the smart speakers. Further work is needed to expand data collection efforts beyond self-reports, which would offer additional perspectives on the acceptability of smart speakers in intervention research and provide greater detail on evaluation criteria, such as fidelity.

Finally, interviews at T1 and T2 were only conducted with participants still enrolled in the study at those time points. While data were collected from individuals who did not adhere to the intervention, it did not include input from those individuals who did not complete the study. This factor limits the interpretability of these findings, as input from noncompleters may have included perceptions related to a lack of feasibility or acceptability of smart speakers for use in a PA program.

Conclusions

In conclusion, findings from this study found smart speakers to be acceptable and appropriate for PA intervention research involving older adults, with participants indicating a willingness to adopt these delivery mechanisms for the delivery of the intervention program as well as for their everyday use. However, the feasibility of these devices for use in research contexts was mixed as they require specific and specialized attention to technical support and troubleshooting when used with older adults. Finally, applications developed to run on smart speakers must be developed to minimize disruption, whether because of flaws in design or through careful planning related to the overall Wi-Fi infrastructure, as weakness in this capacity may impair the ability of smart speakers to deliver interventions with high fidelity.

This study contributes to intervention research in that it evaluates the acceptability and feasibility of smart speakers as a behavioral intervention delivery infrastructure or the mechanisms through which an intervention is delivered, separately and distinctively from the technology that comprises an intervention, which here was a PA program application designed to enhance the physical well-being of sedentary older adults. Conducting separate evaluations of these intervention delivery elements is necessary to ensure a thorough assessment of intervention outcomes. Results from this study highlighted that older adults perceive smart speakers to be useful and easy to use. Future studies might explore the suitability of smart speakers as a delivery infrastructure for aspects of behavioral interventions requiring smart speaker functionalities, such as the setting or reminders or the streaming of media content. Research on the use of smart speakers in other specialized populations, such as those with visual impairment or limited mobility, may also prove fruitful.

In addition, these findings offer important insight for research practitioners. At the very basic level, it cautions against oversimplifying the implications of using complex delivery infrastructures, especially with a population such as older adults that might lag the general population in the adoption and use of emerging technologies. On one level, such oversimplification may overlook important aspects of technological delivery mechanisms, such as the provision of technical support and troubleshooting, which can often tap into limited research resources. On a more granular level, the same functional and technological issues that prompt the need for support resources, such as ensuring continuous Wi-Fi connectivity, can ultimately negatively impact intervention fidelity and compromise the integrity of the research process. Taken together, smart speakers offer a novel delivery infrastructure for behavioral intervention research but also require careful planning.

Acknowledgments

This work was supported by the University of Illinois Chicago Center for Clinical and Translational Science; the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1TR002003; and the National Institute on Aging of the National Institutes of Health through grant R24AG064191. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

Activities of the physical activity program.

Measures of technology acceptability and feasibility.

Interview guide.

• Berkowsky RW, Czaja SJ. The use of technology in behavioral intervention research: advantages and challenges. In: Behavioral Intervention Research: Designing, Evaluating, and Implementing. Berlin, Germany. Springer; Dec 2015.
• Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;35(4):332-338. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Portet F, Vacher M, Golanski C, Roux C, Meillon B. Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects. Pers Ubiquit Comput. Oct 2, 2011;17:127-144. [ CrossRef ]
• Sunshine J. Smart speakers: the next frontier in mHealth. JMIR Mhealth Uhealth. Feb 21, 2022;10(2):e28686. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sezgin E, Militello LK, Huang Y, Lin S. A scoping review of patient-facing, behavioral health interventions with voice assistant technology targeting self-management and healthy lifestyle behaviors. Transl Behav Med. Aug 07, 2020;10(3):606-628. [ CrossRef ] [ Medline ]
• Marsch L, Lord S, Dallery J. Behavioral Health Care and Technology: Using Science-Based Innovations to Transform Practice. New York, NY. Oxford University Press; 2014.
• Hermes ED, Lyon AR, Schueller SM, Glass JE. Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes. J Med Internet Res. Jan 25, 2019;21(1):e11752. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hoy MB. Alexa, Siri, Cortana, and more: an introduction to voice assistants. Med Ref Serv Q. 2018;37(1):81-88. [ CrossRef ] [ Medline ]
• Choi Y, Demiris G, Thompson H. Feasibility of smart speaker use to support aging in place. Innov Aging. Nov 11, 2018;2(suppl_1):560. [ CrossRef ]
• Orr DA, Sanchez L. Alexa, did you get that? Determining the evidentiary value of data stored by the Amazon® Echo. Digit Investig. Mar 2018;24:72-78. [ CrossRef ]
• Gray S. Always on: privacy implications of microphone-enabled devices. Future of Privacy Forum. Apr 2016. URL: https://fpf.org/wp-content/uploads/2016/04/FPF_Always_On_WP.pdf [accessed 2023-11-02]
• Gao C, Chandrasekaran V, Fawaz K, Banerjee S. Traversing the quagmire that is privacy in your smart home. In: Proceedings of the 2018 Workshop on IoT Security and Privacy. 2018. Presented at: IoT S&P '18; August 20, 2018; Budapest, Hungary. [ CrossRef ]
• Pfeifle A. Alexa, what should we do about privacy? Protecting privacy for users of voice-activated devices. Wash Law Rev. 2018;93(1):421.
• Amazon.com: life of an utterance. Amazon. URL: https://www.amazon.com/b/?node=23608618011 [accessed 2023-11-03]
• Fischer SH, David D, Crotty BH, Dierks M, Safran C. Acceptance and use of health information technology by community-dwelling elders. Int J Med Inform. Sep 2014;83(9):624-635. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hargittai E, Piper AM, Morris MR. From internet access to internet skills: digital inequality among older adults. Univ Access Inf Soc. May 3, 2018;18:881-890. [ CrossRef ]
• How popular is internet use among older people? Eurostat. May 17, 2021. URL: https://ec.europa.eu/eurostat/web/products-eurostat-news/-/edn-20210517-1 [accessed 2024-05-22]
• Internet, broadband fact sheet. Pew Research Center. Jan 31, 2024. URL: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/ [accessed 2024-05-22]
• Cotten SR. Technologies and aging: understanding use, impacts, and future needs. In: Handbook of Aging and the Social Sciences (Ninth Edition). Cambridge, MA. Academic Press; 2021.
• Rikard RV, Berkowsky RW, Cotten SR. Discontinued information and communication technology usage among older adults in continuing care retirement communities in the United States. Gerontology. 2018;64(2):188-200. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rosenberg L, Kottorp A, Winblad B, Nygård L. Perceived difficulty in everyday technology use among older adults with or without cognitive deficits. Scand J Occup Ther. 2009;16(4):216-226. [ CrossRef ] [ Medline ]
• Bobillier Chaumon ME, Michel C, Tarpin Bernard F, Croisile B. Can ICT improve the quality of life of elderly adults living in residential home care units? From actual impacts to hidden artefacts. Behav Inf Technol. Oct 29, 2013;33(6):574-590. [ CrossRef ]
• Mynatt ED, Rogers WA. Developing technology to support the functional independence of older adults. Ageing Int. Dec 2001;27:24-41. [ CrossRef ]
• Robinson L, Cotten SR, Ono H, Quan-Haase A, Mesch G, Chen W, et al. Digital inequalities and why they matter. Inf Commun Soc. Mar 16, 2015;18(5):569-582. [ CrossRef ]
• McCloud R, Perez C, Bekalu MA, Viswanath K. Using smart speaker technology for health and well-being in an older adult population: pre-post feasibility study. JMIR Aging. May 09, 2022;5(2):e33498. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ziman R, Walsh G. Factors affecting seniors' perceptions of voice-enabled user interfaces. In: Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 2018. Presented at: CHI EA '18; April 21-26, 2018; Montreal, QC. [ CrossRef ]
• Kowalski J, Jaskulska A, Skorupska K, Abramczuk K, Biele C, Kope? W, et al. Older adults and voice interaction: a pilot study with Google home. In: Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 2019. Presented at: CHI EA '19; May 4-9, 2019; Glasgow, UK. [ CrossRef ]
• Edwards KJ, Jones RB, Shenton D, Page T, Maramba I, Warren A, et al. The use of smart speakers in care home residents: implementation study. J Med Internet Res. Dec 20, 2021;23(12):e26767. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim S, Choudhury A. Exploring older adults’ perception and use of smart speaker-based voice assistants: a longitudinal study. Comput Hum Behav. Nov 2021;124:106914. [ CrossRef ]
• Martín-García AV, Redolat R, Pinazo-Hernandis S. Factors influencing intention to technological use in older adults. The TAM model aplication. Res Aging. 2022;44(7-8):573-588. [ CrossRef ] [ Medline ]
• Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q. Sep 2003;27(3):425-478. [ CrossRef ]
• Nied RJ, Franklin B. Promoting and prescribing exercise for the elderly. Am Fam Physician. Feb 01, 2002;65(3):419-426. [ FREE Full text ] [ Medline ]
• World report on ageing and health. World Health Organization. Sep 29, 2015. URL: https://www.who.int/publications/i/item/9789241565042 [accessed 2023-11-04]
• Harvey JA, Chastin SF, Skelton DA. Prevalence of sedentary behavior in older adults: a systematic review. Int J Environ Res Public Health. Dec 02, 2013;10(12):6645-6661. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spiteri K, Broom D, Bekhet AH, de Caro JX, Laventure B, Grafton K. Barriers and motivators of physical activity participation in middle-aged and older-adults - a systematic review. J Aging Phys Act. Sep 01, 2019;27(4):929-944. [ CrossRef ] [ Medline ]
• Buman MP, Daphna Yasova L, Giacobbi PRJ. Descriptive and narrative reports of barriers and motivators to physical activity in sedentary older adults. Psychol Sport Exerc. May 2010;11(3):223-230. [ CrossRef ]
• Cress ME, Buchner DM, Prohaska T, Rimmer J, Brown M, Macera C, et al. Best practices for physical activity programs and behavior counseling in older adult populations. J Aging Phys Act. Jan 2005;13(1):61-74. [ CrossRef ] [ Medline ]
• Bethancourt HJ, Rosenberg DE, Beatty T, Arterburn DE. Barriers to and facilitators of physical activity program use among older adults. Clin Med Res. Sep 2014;12(1-2):10-20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chin J, Quinn K, Muramatsu N, Marquez D. A user study on the feasibility and acceptance of delivering physical activity programs to older adults through conversational agents. Proc Hum Factors Ergon Soc Annu Meet. Dec 2020;64(1):33-37. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ashworth NL, Chad KE, Harrison EL, Reeder BA, Marshall SC. Home versus center based physical activity programs in older adults. Cochrane Database Syst Rev. Jan 25, 2005;2005(1):CD004017. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cobiac LJ, Vos T, Barendregt JJ. Cost-effectiveness of interventions to promote physical activity: a modelling study. PLoS Med. Jul 14, 2009;6(7):e1000110. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of web-based vs. non-web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res. Nov 10, 2004;6(4):e40. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Valle C, Tate D. Technology-based interventions to promote diet, exercise, and weight control. In: Behavioral Health Care and Technology: Using Science-Based Innovations to Transform Practice. Oxford, UK. Oxford University Press; Nov 2014.
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 2011;38(2):65-76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guzman AL. Voices in and of the machine: source orientation toward mobile virtual assistants. Comput Hum Behav. Jan 2019;90:343-350. [ CrossRef ]
• Foehr J, Germelmann CC. Alexa, can I trust you? Exploring consumer paths to trust in smart voice-interaction technologies. J Assoc Consum Res. Apr 2020;5(2):181-205. [ CrossRef ]
• Taherdoost H. A review of technology acceptance and adoption models and theories. Procedia Manuf. 2018;22:960-967. [ CrossRef ]
• Bixter MT, Blocker KA, Mitzner TL, Prakash A, Rogers WA. Understanding the use and non-use of social communication technologies by older adults: a qualitative test and extension of the UTAUT model. Gerontechnology. Jun 2019;18(2):70-88. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Magsamen-Conrad K, Upadhyaya S, Joa CY, Dowd J. Bridging the divide: using UTAUT to predict multigenerational tablet adoption practices. Comput Human Behav. Sep 01, 2015;50:186-196. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rouidi M, Elouadi AE, Hamdoune A, Choujtani K, Chati A. TAM-UTAUT and the acceptance of remote healthcare technologies by healthcare professionals: a systematic review. Inform Med Unlocked. 2022;32:101008. [ CrossRef ]
• Moxley J, Sharit J, Czaja SJ. The factors influencing older adults' decisions surrounding adoption of technology: quantitative experimental study. JMIR Aging. Nov 23, 2022;5(4):e39890. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Berkowsky RW, Sharit J, Czaja SJ. Factors predicting decisions about technology adoption among older adults. Innov Aging. Feb 21, 2018;2(1):igy002. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim D. Can healthcare apps and smart speakers improve the health behavior and depression of older adults? A quasi-experimental study. Front Digit Health. Feb 23, 2023;5:1117280. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yan T, Wilber KH, Wieckowski J, Simmons WJ. Results from the healthy moves for aging well program: changes of the health outcomes. Home Health Care Serv Q. 2009;28(2-3):100-111. [ CrossRef ] [ Medline ]
• The trusted standard in research, ethics, compliance, and safety training. CITI Program. URL: https://about.citiprogram.org/ [accessed 2024-05-20]
• Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. Dec 2007;19(6):349-357. [ CrossRef ] [ Medline ]
• Heerink M, Kröse BJ, Vanessa E, Wielinga BJ. The influence of social presence on acceptance of a companion robot by older people. J Phys Agents. Jan 2008;2(2):33-40. [ CrossRef ]
• Rosenberg L, Nygård L, Kottorp A. Everyday technology use questionnaire: psychometric evaluation of a new assessment of competence in technology use. Occup Ther J Res. 2009;29(2):52-62. [ CrossRef ]
• Laver K, George S, Ratcliffe J, Crotty M. Measuring technology self efficacy: reliability and construct validity of a modified computer self efficacy scale in a clinical rehabilitation setting. Disabil Rehabil. 2012;34(3):220-227. [ CrossRef ] [ Medline ]
• Pfeiffer E. A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients. J Am Geriatr Soc. Oct 1975;23(10):433-441. [ CrossRef ] [ Medline ]
• Roccaforte WH, Burke WJ, Bayer BL, Wengel SP. Reliability and validity of the short portable mental status questionnaire administered by telephone. J Geriatr Psychiatry Neurol. 1994;7(1):33-38. [ Medline ]
• Happ MB, Dabbs AD, Tate J, Hricik A, Erlen J. Exemplars of mixed methods data combination and analysis. Nurs Res. 2006;55(2 Suppl):S43-S49. [ CrossRef ] [ Medline ]
• Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal. 1989;11(3):255-274. [ CrossRef ]
• Wi-Fi networks that are incompatible or aren’t recommended. Google Nest Help. URL: https://support.google.com/googlenest/answer/9249905?hl=en [accessed 2023-11-15]
• Chang F, Sheng L, Gu Z. Investigating the integration and the long-term use of smart speakers in older adults' daily practices: qualitative study. JMIR Mhealth Uhealth. Feb 12, 2024;12:e47472. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Chin J. OK Google, let's learn: using voice user interfaces for informal self-regulated learning of health topics among younger and older adults. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. Presented at: CHI '23; April 23-28, 2023; Hamburg, Germany. [ CrossRef ]
• Desai S, Lundy M, Chin J. “A painless way to learn:” designing an interactive storytelling voice user interface to engage older adults in informal health information learning. In: Proceedings of the 5th International Conference on Conversational User Interfaces. 2023. Presented at: CUI '23; July 19-21, 2023; Eindhoven, The Netherlands. [ CrossRef ]
• Thirunavukarasu AJ, Ting DS, Elangovan K, Gutierrez L, Tan TF, Ting DS. Large language models in medicine. Nat Med. Aug 2023;29(8):1930-1940. [ CrossRef ] [ Medline ]
• Manikonda L, Deotale A, Kambhampati S. What's up with privacy?: user preferences and privacy concerns in intelligent personal assistants. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018. Presented at: AIES '18; February 2-3, 2018; New Orleans, LA. [ CrossRef ]
• Edu JS, Such JM, Suarez-Tangil G. Smart home personal assistants: a security and privacy review. ACM Comput Surv. Dec 06, 2020;53(6):1-36. [ CrossRef ]
• Lutz C, Newlands G. Privacy and smart speakers: a multi-dimensional approach. Inf Soc. Mar 27, 2021;37(3):147-162. [ CrossRef ]

Abbreviations

Edited by E Wethington, A Dominello, I Kronish, J Kaye; submitted 22.11.23; peer-reviewed by S Muraki, M Chatzimina, H Li; comments to author 05.04.24; revised version received 01.06.24; accepted 29.06.24; published 30.08.24.

©Kelly Quinn, Sarah Leiser Ransom, Carrie O'Connell, Naoko Muramatsu, David X Marquez, Jessie Chin. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 30.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Open access
• Published: 26 August 2024

Solid health care waste management practice in Ethiopia, a convergent mixed method study

• Yeshanew Ayele Tiruneh 1 ,
• L. M. Modiba 2 &
• S. M. Zuma 2  

BMC Health Services Research volume  24 , Article number:  985 ( 2024 ) Cite this article

66 Accesses

Metrics details

Introduction

Healthcare waste is any waste generated by healthcare facilities that is considered potentially hazardous to health. Solid healthcare waste is categorized into infectious and non-infectious wastes. Infectious waste is material suspected of containing pathogens and potentially causing disease. Non-infectious waste includes wastes that have not been in contact with infectious agents, hazardous chemicals, or radioactive substances, similar to household waste, i.e. plastic, papers and leftover foods.

This study aimed to investigate solid healthcare waste management practices and develop guidelines to improve solid healthcare waste management practices in Ethiopia. The setting was all health facilities found in Hossaena town.

A mixed-method study design was used. For the qualitative phase of this study, eight FGDs were conducted from 4 government health facilities, one FGD from each private health facility (which is 37 in number), and forty-five FGDs were conducted. Four FGDs were executed with cleaners; another four were only health care providers because using homogeneous groups promotes discussion. The remaining 37 FGDs in private health facilities were mixed from health professionals and cleaners because of the number of workers in the private facilities. For the quantitative phase, all health facilities and health facility workers who have direct contact with healthcare waste management practice participated in this study. Both qualitative and quantitative study participants were taken from the health facilities found in Hossaena town.

Seventeen (3.1%) health facility workers have hand washing facilities. Three hundred ninety-two (72.6%) of the participants agree on the availability of one or more personal protective equipment (PPE) in the facility ‘‘ the reason for the absence of some of the PPEs, like boots and goggles, and the shortage of disposable gloves owes to cost inflation from time to time and sometimes absent from the market’’ . The observational finding shows that colour-coded waste bins are available in 23 (9.6%) rooms. 90% of the sharp containers were reusable, and 100% of the waste storage bins were plastic buckets that were easily cleanable. In 40 (97.56%) health facilities, infectious wastes were collected daily from the waste generation areas to the final disposal points. Two hundred seventy-one (50.2%) of the respondents were satisfied or agreed that satisfactory procedures are available in case of an accident. Only 220 (40.8%) respondents were vaccinated for the Hepatitis B virus.

Hand washing facilities, personal protective equipment and preventive vaccinations are not readily available for health workers. Solid waste segregation practices are poor and showed that solid waste management practices (SWMP) are below the acceptable level.

Peer Review reports

Healthcare waste (HCW) encompasses all types of waste generated while providing health-related services, spanning activities such as diagnosis, immunization, treatment, and research. It constitutes a diverse array of materials, each presenting potential hazards to health and the environment. Within the realm of HCW, one finds secretions and excretions from humans, cultures, and waste containing a stock of infectious agents. Discarded plastic materials contaminated with blood or other bodily fluids, pathological wastes, and discarded medical equipment are classified as healthcare waste. Sharps, including needles, scalpels, and other waste materials generated during any healthcare service provision, are also considered potentially hazardous to health [ 1 ].

Healthcare waste in solid form (HCW) is commonly divided into two primary groups: infectious and non-infectious. The existence of pathogens in concentrations identifies infectious waste or amounts significant enough to induce diseases in vulnerable hosts [ 1 ] If healthcare facility waste is free from any combination with infectious agents, nearly 85% is categorized as non-hazardous waste, exhibiting characteristics similar to conventional solid waste found in households [ 2 ]. World Health Organization (WHO) recommends that appropriate colour-coded waste receptacles be available in all medical and other waste-producing areas [ 3 ].

Solid waste produced in the course of healthcare activities carries a higher potential for infection and injury than any other type of waste. Improper disposal of sharps waste increases the risk of disease transmission among health facility workers and general populations [ 1 ]. Inadequate and inappropriate handling of healthcare waste may have serious public health consequences and a significant environmental impact. The World Health Organization (2014) guidelines also include the following guidance for hand washing and the use of alcohol-based hand rubs: Wash hands before starting work, before entering an operating theatre, before eating, after touching contaminated objects, after using a toilet, and in all cases where hands are visibly soiled [ 4 ].

Among the infectious waste category, sharps waste is the most hazardous waste because of its ability to puncture the skin and cause infection [ 3 ]. Accidents or occurrences, such as near misses, spills, container damage, improper waste segregation, and incidents involving sharps, must be reported promptly to the waste management officer or an assigned representative [ 5 ].

Africa is facing a growing waste management crisis. While the volumes of waste generated in Africa are relatively small compared to developed regions, the mismanagement of waste in Africa already impacts human and environmental health. Infectious waste management has always remained a neglected public health problem in developing countries, resulting in a high burden of environmental pollution affecting the general masses. In Ethiopia, there is no updated separate regulation specific to healthcare waste management in the country to enforce the proper management of solid HCW [ 6 ].

In Ethiopia, like other developing countries, healthcare waste segregation practice was not given attention and did not meet the minimum HCWM standards, and it is still not jumped from paper. Previous study reveals that healthcare waste generation rates are significantly higher than the World Health Organization threshold, which ranges from 29.5–53.12% [ 7 , 8 ]. In Meneilk II Hospital, the proportion of infectious waste was 53.73%, and in the southern and northern parts of Ethiopia, it was 34.3 and 53%, respectively. Generally, this figure shows a value 3 to 4 times greater than the threshold value recommended by the World Health Organization [ 7 ].

Except for sharp wastes, segregation practice was poor, and all solid wastes were collected without respecting the colour-coded waste disposal system [ 9 ]. The median waste generation rate was found to vary from 0.361- 0.669 kg/patient/day, comprising 58.69% non-hazardous and 41.31% hazardous wastes. The amount of waste generated increased as the number of patients flow increased. Public hospitals generated a high proportion of total healthcare waste (59.22%) in comparison with private hospitals (40.48) [ 10 ]. The primary SHCW treatment and disposal mechanism was incineration, open burning, burring into unprotected pits and open dumping on municipal dumping sites as well as in the hospital backyard. Carelessness, negligence of the health workers, patients and cleaners, and poor commitment of the facility leaders were among the major causes of poor HCWM practice in Ethiopia [ 9 ]. This study aimed to investigate solid healthcare waste management practices and develop guidelines to improve solid healthcare waste management practices in Ethiopia.

The setting for this study was all health facilities found in Hossaena town, which is situated 232 kms from the capital city of Ethiopia, Addis Ababa, and 165 kms from the regional municipality of Hawasa. The health facilities found in the town were one university hospital, one private surgical centre, three government health centres, 17 medium clinics, and 19 small clinics were available in the city and; health facility workers who have direct contact with generating and disposal of HCW and those who are responsible as a manager of health facilities found in Hossaena town are the study settings. All health facilities except drug stores and health facility workers who have direct contact with healthcare waste generation participated in this study.

A mixed-method study design was used. For the quantitative part of this study, all healthcare workers who have direct contact with healthcare waste management practice participated in this study, and one focus group discussion from each health facility was used. Both of the study participants were taken from the same population. All health facility workers who have a role in healthcare waste management practice were included in the quantitative part of this study. The qualitative data collection phase used open-ended interviews, focus group discussions, and visual material analysis like posters and written materials. All FGDs were conducted by the principal investigator, one moderator, and one note-taker, and it took 50 to 75 min. 4–6 participants participated in each FGD.

According to Elizabeth (2018: 5), cited by Creswell and Plano (2007: 147), the mixed method is one of the research designs with philosophical assumptions as well as methods of inquiry. As a method, it focuses on collecting, analyzing, and mixing both quantitative and qualitative data in a single study. As a methodology, it involves philosophical assumptions guiding the direction of the collection and analysis and combining qualitative and quantitative approaches in many phases of the research project. The central premise is that using qualitative and quantitative approaches together provides a better understanding of the research problems than either approach alone.

The critical assumption of the concurrent mixed methods approach in this study is that quantitative and qualitative data provide different types of information, often detailed views of participants’ solid waste management practice qualitatively and scores on instruments quantitatively, and together, they yield results that should be the same. In this approach, the researcher collected quantitative and qualitative data almost simultaneously and analyzed them separately to cross-validate or compare whether the findings were similar or different between the qualitative and quantitative information. Concurrent approaches to the data collection process are less time-consuming than other types of mixed methods studies because both data collection processes are conducted on time and at the same visit to the field [ 11 ].

Data collection

The data collection involves collecting both quantitative and qualitative data simultaneously. The quantitative phase of this study assessed three components. Health care waste segregation practice, the availability of waste segregation equipment for HCW segregation, temporary storage facilities, transportation for final disposal, and disposal facilities data were collected using a structured questionnaire and observation of HCW generation. Recycling or re-using practice, waste treatment, the availability of the HCWM committee, and training data were collected.

Qualitative data collection

The qualitative phase of the data collection for this study was employed by using focus group discussions and semi-structured interviews about SHCWMP. Two focus group discussions (FGD) from each health facility were conducted in the government health facilities, one at the administrative level and one at the technical worker level, and one FGD was conducted for all private health facilities because of the number of available health facility workers. Each focus group has 4–6 individuals.

In this study, the qualitative and the quantitative data provide different information, and it is suitable for this study to compare and contrast the findings of the two results to obtain the best understanding of this research problem.

Quantitative data collection

The quantitative data were entered into Epi data version 3.1 to minimize the data entry mistakes and exported to the statistical package for social science SPSS window version 27.0 for analysis. A numeric value was assigned to each response in a database, cleaning the data, recoding, establishing a codebook, and visually inspecting the trends to check whether the data were typically distributed.

Data analysis

Data were analyzed quantitatively by using relevant statistical tools, such as SPSS. Descriptive statistics and the Pearson correlation test were used for the bivariate associations and analysis of variance (ANOVA) to compare the HCW generation rate between private and government health facilities and between clinics, health centres and hospitals in the town. Normality tests were performed to determine whether the sample data were drawn from a normally distributed population.

The Shapiro–Wilk normality tests were used to calculate a test statistic based on the sample data and compare it to critical values. The Shapiro–Wilk test is a statistical test used to assess whether a given sample comes from a normally distributed population. The P value greater than the significance level of 0.05 fails to reject the null hypothesis. It concludes that there is not enough evidence to suggest that the data does not follow the normal distribution. Visual inspection of a histogram, Q-Q plot, and P-P plot (probability-probability plot) was assessed.

Bivariate (correlation) analysis assessed the relationships between independent and dependent variables. Then, multiple linear regression analysis was used to establish the simple correlation matrices between different variables for investigating the strength relationships of the study variables in the analysis. In most variables, percentages and means were used to report the findings with a 95% confidence interval. Open-ended responses and focused group findings were undertaken by quantifying and coding the data to provide a thematic narrative explanation.

Appropriate and scientific care was taken to maintain the data quality before, during, and after data collection by preparing the proper data collection tools, pretesting the data collection tools, providing training for data collectors, and proper data entry practice. Data were cleaned on a daily basis during data collection practice, during data entry, and before analysis of its completeness and consistency.

Data analysis in a concurrent design consists of three phases. First, analyze the quantitative database in terms of statistical results. Second, analyze the qualitative database by coding the data and collapsing the codes into broad themes. Third comes the mixed-method data analysis. This is the analysis that consists of integrating the two databases. This integration consists of merging the results from both the qualitative and the quantitative findings.

Descriptive analysis was conducted to describe and summarise the data obtained from the samples used for this study. Reliability statistics for constructs, means and modes of each item, frequencies and percentage distributions, chi-square test of association, and correlations (Spearman rho) were used to portray the respondents’ responses.

All patient care-providing health facilities were included in this study, and the generation rate of healthcare waste and composition assessed the practice of segregation, collection, transportation, and disposal system was observed quantitatively using adopted and adapted structured questionnaires. To ensure representativeness, various levels of health facilities like hospitals, health centres, medium clinics, small clinics and surgical centres were considered from the town. All levels of health facilities are diagnosing, providing first aid services and treating patients accordingly.

The hospital and surgical centre found in the town provide advanced surgical service, inpatient service and food for the patients that other health facilities do not. The HCW generation rate was proportional to the number of patients who visited the health facilities and the type of service provided. The highest number of patients who visited the health facilities was in NEMMCSH; the service provided was diverse, and the waste generation rate was higher than that of other health facilities. About 272, 18, 15, 17, and 20 average patients visited the health facilities daily in NEMMCSH: government health centres, medium clinics, small clinics, and surgical centres. Paper and cardboard (141.65 kg), leftover food (81.71 kg), and contaminated gloves (42.96 kg) are the leading HCWs generated per day.

A total of 556 individual respondents from sampled health facilities were interviewed to complete the questionnaire. The total number of filled questionnaires was 540 (97.1) from individuals representing these 41 health facilities.

The principal investigator observed the availability of handwashing facilities near SHCW generation sites. 17(3.1%) of health facility workers had hand washing facilities near the health care waste generation and disposal site. Furthermore,10 (3.87%), 2 (2.1%), 2 (2.53%), 2 (2.1%), 1 (6.6%) of health facility workers had the facility of hand washing near the health care waste generation site in Nigist Eleni Mohamed Memorial Comprehensive Specialized Hospital (NEMMCSH), government health centres, medium clinics, small clinics, and surgical centre respectively. This finding was nearly the same as the study findings conducted in Myanmar; the availability of hand washing facilities near the solid health care waste generation was absent in all service areas [ 12 ]. The observational result was convergent with the response of facility workers’ response regarding the availabilities of hand washing facilities near to the solid health care waste generation sites.

The observational result was concurrent with the response of facility workers regarding the availability of hand-washing facilities near the solid health care waste generation sites.

The availability of personal protective equipment (PPE) was checked in this study. Three hundred ninety-two (72.6%) of the respondents agree on the facility’s availability of one or more personal protective equipment (PPE). The availability of PPEs in different levels of health facilities shows 392 (72.6%), 212 (82.2%), 56 (58.9%), 52 (65.8%), 60 (65.2%), 12 (75%) health facility workers in NEMMCSH, government health centres, medium clinics, small clinics, and surgical centres respectively agree to the presence of personal protective equipment in their department. The analysis further shows that the availability of masks for healthcare workers was above the mean in NEMMCSH and surgical centres.

Focus group participants indicated that health facilities did not volunteer to supply Personal protective equipment (PPEs) for the cleaning staff.

“We cannot purchase PPE by ourselves because of the salary paid for the cleaning staff.”

Cost inflation and the high cost of purchasing PPEs like gloves and boots are complained about by all (41) health facility owners.

“the reason for the absence of some of the PPEs like boots, goggles, and shortage of disposable gloves are owing to cost inflation from time to time and sometimes absent from the market is the reason why we do not supply PPE to our workers.”

Using essential personal protective equipment (PPEs) based on the risk (if the risk is a splash of blood or body fluid, use a mask and goggles; if the risk is on foot, use appropriate shoes) is recommended by the World Health Organization [ 13 ]. The mean availability of gloves in health facilities was 343 (63.5% (95% CI: 59.3–67.4). Private health institutions are better at providing gloves for their workers, 67.1%, 72.8%, and 62.5% in medium clinics, small clinics, and surgical centres, respectively, which is above the mean.

Research participants agree that.

‘‘ there is a shortage of gloves to give service in Nigist Eleni Mohamed Memorial Comprehensive Specialized Hospital (NEMMCSH) and government health centres .’’

Masks are the most available personal protective equipment for health facility workers compared to others. 65.4%, 55.6%, and 38% of the staff are available with gloves, plastic aprons and boots, respectively.

The mean availability of masks, heavy-duty gloves, boots, and aprons was 71.1%, 65.4%, 38%, and 44.4% in the study health facilities. Health facility workers were asked about the availability of different personal protective equipment, and 38% of the respondents agreed with the presence of boots in the facility. Still, the qualitative observational findings of this study show that all health facility workers have no shoes or footwear during solid health care waste management practice.

SHCW segregation practice was checked by observing the availability of SHCW collection bins in each patient care room. Only 4 (1.7%) of the room’s SHCW bins are collected segregated (non-infectious wastes segregated in black bins and infectious wastes segregated in yellow bins) based on the World Health Organization standard. Colour-coded waste bins, black for non-infectious and yellow for infectious wastes, were available in 23 (9.6%) rooms. 90% of the sharp containers were reusable, and 100% of the waste storage bins were plastic buckets that were easily cleanable. Only 6.7% of the waste bins were pedal operated and adequately covered, and the rest were fully opened, or a tiny hole was prepared on the container’s cover. All of the healthcare waste disposal bins in each health facility and at all service areas were away from the arm’s reach distance of the waste generation places, and this is contrary to World Health Organization SHCWM guidelines [ 13 ]. The observation result reveals that the reason for the above result was that medication trolleys were not used during medication or while healthcare providers provided any health services to patients.

Most medical wastes are incinerated. Burning solid and regulated medical waste generated by health care creates many problems. Medical waste incinerators emit toxic air pollutants and ash residues that are the primary source of environmental dioxins. Public concerns about incinerator emissions and the creation of federal regulations for medical waste incinerators are causing many healthcare facilities to rethink their choices in medical waste treatment. Health Care Without Harm [ 14 ], states that non-incineration treatment technologies are a growing and developing field. The U.S. National Academy of Science 2000 argued that the emission of pollutants during incineration is a potential risk to human health, and living or working near an incineration facility can have social, economic, and psychological effects [ 15 ].

The incineration of solid healthcare waste technology has been accepted and adopted as an effective method in Ethiopia. Incineration of healthcare waste can produce secondary waste and pollutants if the treatment facilities are not appropriately constructed, designed, and operated. It can be one of the significant sources of toxic substances, such as polychlorinated dibenzo-dioxins/dibenzofurans (PCDD/ PCDF), polyvinyl chloride (PVC), hexachlorobenzenes and polychlorinated biphenyls, and dioxins and furans that are known as hazardous pollutants. These pollutants may have undesirable environmental impacts on human and animal health, such as liver failure and cancer [ 15 , 16 ].

All government health facilities (4 in number) used incineration to dispose of solid waste. 88.4% and 100% of the wastes are incinerated in WUNEMMCSH and government health centres. This finding contradicts the study findings in the United States of America and Malaysia, in which 49–60% and 59–60 were incinerated, respectively, and the rest were treated using other technologies [ 15 , 16 ].

World Health Organization (2014:45) highlighted those critical elements of the appropriate operation of incinerators include effective waste reduction and waste segregation, placing incinerators away from populated areas, satisfactory engineered design, construction following appropriate dimensional plans, proper operation, periodic maintenance, and staff training and management are mandatory.

Solid waste collection times should be fixed and appropriate to the quantity of waste produced in each area of the health care facility. General waste should not be collected simultaneously or in the same trolley as infectious or hazardous wastes. The collection should be done daily for most wastes, with collection timed to match the pattern of waste generation during the day [ 13 ].

SHCW segregation practices were observed for 240 rooms in 41 health facilities that provide health services in the town. In government health centres, medium clinics, small clinics, and surgical centres, SHCW segregation practice was not based on the World Health Organization standard. All types of solid waste were collected in a single container near the generation area, and there were no colour-coded SHCW storage dust bins. Still, in NEMMCSH, in most of the service areas, colour-coded waste bins are available, and the segregation practice was not based on the standard. Only 3 (10%) of the dust bins collected the appropriate wastes according to the World Health Organization standard, and the rest were mixed with infectious and non-infectious SHCW.

Table 1 below shows health facility managers were asked about healthcare waste segregation practices, and 9 (22%) of the facility leaders responded that there is an appropriate solid healthcare waste segregation practice in their health facilities. Still, during observation, only 4 (1.7%) of the rooms in two (4.87%) of the facilities, SHCW bins collected the segregated wastes (non-infectious wastes segregated at the black bin and infectious wastes segregated at yellow bin) based on the world health organization standard. The findings of this study show there is a poor segregation practice, and all kinds of solid wastes are collected together.

In 40 (97.56%) health facilities, infectious wastes were collected daily from the waste generation areas to the final disposal points. During observation in one of the study health facilities, infectious wastes were not collected daily and left for days. Utility gloves, boots, and aprons are not available for cleaning staff to collect and transport solid healthcare wastes in all study health facilities. 29.26% of the facilities’ cleaning staff have a face mask, and 36.5% of the facilities remove waste bins from the service area when 3/4 full, and the rest were not removed or replaced with new ones. There is a separate container only in 2 health facilities for infectious and non-infectious waste segregation practice, and the rest were segregated and collected using single and non-colour coded containers.

At all of the facilities in the study area, SHCW was transported from the service areas to the disposal site were transported manually by carrying the collection container and there is no trolley for transportation. This finding was contrary to the study findings conducted in India, which show segregated waste from the generation site was being transported through the chute to the carts placed at various points on the hospital premises by skilled sanitary workers [ 17 ].

Only 2 out of 41 health facilities have temporary solid waste storage points at the facility. One of the temporary storage places was clean, and the other needed to be properly cleaned and unsightly. Two (100%) of the temporary storage areas are not fenced and have no restriction to an authorized person. Temporary storage areas are available only in two health facilities that are away from the service provision areas.

Observational findings revealed that pre-treatment of SHCW before disposal was not practised at all study health facilities. 95% of the facilities have no water supply for hand washing during and after solid healthcare waste generation, collection, and disposal.

The United States Agency estimated sharp injuries from medical wastes to health professionals and sanitary service personnel for toxic substances and disease registry. Most of the injuries are caused during the recapping of hypodermic needles before disposal into sharps containers [ 13 ]. Nearly half of the respondents, 245 (51.5%), are recapping needles after providing an injection to the patient. Recapping was more practised in NEMMCSH and surgical centres, which is 57.5% and 57.5%, respectively. In government health centres, medium clinics, and surgical centres, the recapping of used needles was practised below the mean, which is 47.9%, 48, and 43.8%, respectively. This finding was reasonable compared to the study findings of Doylo et al. [ 18 ] in western Ethiopia, where 91% of the health workers are recapping needles after injection [ 18 ]. The research finding shows that there is no significant association P-value of 0.82 between the training and recapping of needles after injection.

Focus group participants ’ response for appropriate SHCWMP regarding patients ’ and visitors ’ lack of knowledge on SHCW segregation practice

“The personal responsibilities of patients and visitors on solid HCW disposal should be explained to help appropriate safe waste management practice and maintain good hygiene .” “Providing waste management training and creating awareness are the two aspects of improving SHCW segregation practice.” “Training upgrades and creates awareness on hygiene for all workers.”

Sharp waste collection practices were observed in 240 rooms in the study health facilities, and 9.2% of the rooms used disposable sharp containers.

Sixty per cent (60%), 13.3%, 8.24%, and 15.71% of the sharps containers in NEMMCSH, government health centres, medium clinics, and small clinics, respectively, were using disposable sharps containers; sharps were disposed together with the sharps container, and surgical centre was using reusable sharp collection container. All disposable sharps containers in medium and small clinics used non-puncture-resistant or simple packaging carton boxes. 60% and 13.3% of the disposable sharps containers in NEMMCSH and the government health centre use purposefully manufactured disposable safety boxes.

Needle sticks injury reporting and occurrence

A total of 70 injuries were reported to the health facility manager in the last one year, and 44 of the injuries were reported by health professionals. The rest of the injuries were reported by supportive staff. These injuries were reported from 35 health facilities, and the remaining six health facilities did not report any cases of injury related to work; see Tables 2 and 3 below.

Accidents or incidents, including near misses, spillages, damaged containers, inappropriate segregation, and any incidents involving sharps, should be reported to the waste-management officer. Accidental contamination must be notified using a standard-format document. The cause of the accident or incident should be investigated by the waste-management officer (in case of waste) or another responsible officer, who should also take action to prevent a recurrence [ 13 ]. Two hundred seventy-one (50.2% (CI: 45.7–54.6) of the respondents agree that satisfactory procedures are available in case of an accident, while the remaining 269 (49.8%( CI: 45.4–54.3) of respondents do not agree on the availability of satisfactory procedures in case of an accident, see Table  4 below. The availability of satisfactory procedures in case of an accident is above the mean in medium clinics, which is 60.8%. 132(24.4%) of the staff are pricked by needle stick injury while providing health services. Nearly half of the respondents, 269 (49.8%), who have been exposed to needle stick injury do not get satisfactory procedures after being pricked by a needle, and those who have not been stung by a needle stick injury for the last year. 204 (37.8%) disagree with the presence of satisfactory procedures in the case of a needle stick injury. In NEMMCSH, 30.2% of the research participants were pricked by needle stick injury within one year of period, and 48.8% of those who were stung by needle stick injuries did not agree upon the presence of satisfactory procedures in case of needle stick injuries in the study hospital. 17.9% and 49.5%, 24.1% and 60.8%, 7.6% and 50% of the respondents are pricked by needle sticks, and they disagree on the availability of satisfactory procedures in case of accidents, respectively, in government health centres, medium clinics, small clinics, and surgical centre respectively.

One hundred seventy-seven (32.7% (CI:29.1–37) respondents were exposed to needle stick injury while working in the current health facilities. One hundred three (58.1%) and 26 (32.9%) needle stick injuries were reported from WUNEMMCSH and medium clinics, which is above the mean. One hundred thirty-two(24.7% (95%CI:20.7–28.1) of the respondents are exposed to needle stick injury within one year of the period. Seventy-eight(30.2%), 17 (17.9%), 19 (24.1%), 15 (16.3%), 3 (18.8%) of the staff are injured by needle sticks from NEMMCSH, government health centres, medium clinics, small clinics, and surgical centre staffs respectively within one year of service.

The mean availabilities of satisfactory procedures in case of accidents were 321 (59.4% (CI:55.4–63.7). Out of this, 13.7% of the staff is injured by needle sticks within one year before the survey. Except in NEMMCSH, the mean availabilities of satisfactory procedures were above the mean, which is 50%, 60%, 77.2%, 66.3%, and 81.3% in NEMMCSH, government health centres, medium clinics, small clinics, and surgical centres respectively.

Table 5 below shows that Hepatitis B, COVID-19, and tetanus toxoid vaccinations are the responses of the research participants to an open-ended question on which vaccine they took. The finding shows that 220 (40.8%) of the respondents were vaccinated to prevent themselves from health facility-acquired infection. One hundred fifty-six (70.9%) of the respondents are vaccinated to avoid themselves from Hep B infection. Fifty-nine (26%0.8) of the respondents were vaccinated to protect themselves from two diseases that are Hep B and COVID-19.

Appropriate health care waste management practice was assessed by using 12 questions: availability of colour-coded waste bins, foot-operated dust bins, elbow or foot-operated hand washing basin, personal protective equipment, training, role and responsibility of the worker, the presence of satisfactory procedures in case of an accident, incinerator, vaccination, guideline, onsite treatment, and the availability of poster. The mean of appropriate healthcare waste management practice was 55.58%. The mean of solid health care waste management practice based on the level of health facilities was summed and divided into 12 variables to get each health facility’s level of waste management practice. 64.9%, 45.58%, 49%, 46.9%, and 51.8% are the mean appropriate health care waste management practices in NEMMCSH, government health centres, medium clinics, small clinics, and surgical centres, respectively. In NEMMCSH, the practice of solid healthcare waste management shows above the mean, and the rest was below the mean of solid healthcare waste management practice.

Healthcare waste treatment and disposal practice

Solid waste treatment before disposal was not practised at all study health facilities. There is an incineration practice at all of the study health facilities, and the World Health Organization 2014 recommended three types of incineration practice for solid health care waste management: dual-chamber starved-air incinerators, multiple chamber incinerators, and rotary kilns incinerators. Single-chamber, drum, and brick incinerators do not meet the best available technique requirements of the Stockholm Convention guidelines [ 13 ]. The findings of this study show that none of the incinerators found in the study health facilities meet the minimum standards of solid healthcare waste incineration practice, and they need an air inlet to facilitate combustion. Eleven (26.82%) of the health facilities have an ash pit to dispose of burned SHCW; the majority, 30 (73.17%), dispose of the incinerated ash and burned needles in the municipal waste disposal site. In one out of 11 health facilities with an ash pit, one of the incinerators was built on the ash pit, and the incinerated ashes were disposed of in the ash pit directly. Pre-treatment of SHCW before disposal was not practised at all health facilities; see Table  6 below.

All government health facilities use incineration to dispose of solid waste. 88.4% and 100% of the solid wastes are incinerated in WUNEMMCS Hospital and government health centres, respectively. This finding was not similar to the other studies because other technologies like autoclave microwave and incineration were used for 59–60% of the waste [ 15 ]. Forty-one (100%) of the study facilities were using incinerators, and only 5 (12.19%) of the incinerators were constructed by using brick and more or less promising than others for incinerating the generated solid wastes without considering the emitting gases into the atmosphere and the residue chemicals and minerals in the ashes.

Research participants’ understanding of the environmental friendliness of health care waste management practice was assessed, and the result shows that more than half, 312(57%) of the research participants do not agree on the environmental friendliness of the waste disposal practices in the health facilities. The most disagreement regarding environmental friendliness was observed in NEMMCSH; 100 (38.8%) of the participants only agreed the practice was environmentally friendly of the service. Forty-four (46.3%), 37 (46.8%), 40 (43.5%), and 7 (43.8%) of the participants agree on the environmental friendliness of healthcare waste management practice in government health centres, medium clinics, small clinics, and surgical centres, respectively.

One hundred twenty-five (48.4%) and 39(42.4%) staff are trained in solid health care waste management practice in NEMMCSH and small clinic staff, respectively; this result shows above the mean. Twenty-seven (28.4%), 30 (38%), and 4 (25%) of the staff are trained in health care waste management practice in Government health centres, medium clinics, and surgical centres, respectively. The training has been significantly associated with needle stick injury, and the more trained staff are, the less exposed to needle stick injury. One hundred ninety-six (36.4%) of the participants answered yes to the question about the availability of trainers in the institution. 43.8% of the NEMMCSH staff agreed on the availability of trainers on solid health care waste management, which is above the mean, and 26.3%, 31.6%, 31.5%, and 25% for the government health centres, medium clinics, small clinics, and surgical centre respectively, which is below the mean.

Trained health professionals are more compliant with SHCWM standards, and the self-reported study findings of this study show that 41.7% (95%CI:37.7–46) of the research participants are trained in health care waste management practice. This finding was higher compared to the study findings of Sahiledengle in 2019 in the southeast of Ethiopia, shows 13.0% of healthcare workers received training related to HCWM in the past one year preceding the study period and significantly lower when compared to the study findings in Egypt which is 71% of the study participants were trained on SHCWM [ 8 , 19 , 20 ].

Three out of four government health facility leaders, 17 (45.94%) of private health facility leaders/owners of the clinic and 141 FGD participants complain about the absence of some PPEs like boots and aprons to protect themselves from infectious agents.

‘ ‘Masks, disposable gloves, and changing gowns are a critical shortage at all health facilities.’’

Cleaners in private health facilities are more exposed to infectious agents because of the absence of personal protective equipment. Except for the cleaning staff working in the private surgical centre, all cleaning staff 40 (97.56) of the health facilities complain about the absence of changing gowns and the fact that there are no boots in the facilities.

Cost inflation and the high cost of purchasing PPEs like gloves and boots are complained by all of (41) the health facility owners and the reason for the absence of some of the PPEs like boots, goggles, and shortage of disposable gloves. Sometimes, absence from the market is the reason why we do not supply PPE to our workers.

Thirty-four (82.92%) of the facility leaders are forwarded, and there is a high expense and even unavailability of some of the PPEs, which are the reasons for not providing PPEs for the workers.

‘‘Medical equipment and consumables importers and whole sellers are selective for importing health supplies, and because of a small number of importers in the country and specifically, in the locality, we can’t get materials used for health care waste management practice even disposable gloves. ’’

One of the facility leaders from a private clinic forwarded that before the advent of COVID-19 -19) personal protective equipment was more or less chip-and-get without difficulty. Still, after the advent of the first Japanese COVID-19 patient in Ethiopia, people outside the health facilities collect PPEs like gloves and masks and storing privately in their homes.

‘‘PPEs were getting expensive and unavailable in the market. Incinerator construction materials cost inflation, and the ownership of the facility building are other problems for private health facilities to construct standard incinerators.’’

For all of the focus group discussion participants except in NEMMCSH and two private health facilities, covered and foot-operated dust bins were absent or in a critical shortage compared to the needed ones.

‘‘ Waste bins are open and not colour-coded. The practice attracts flies and other insects. Empty waste bins are replaced without cleaning and disinfecting by using chlorine solution.’’ “HCW containers are not colour-coded, but we are trying to label infectious and non-infectious in Amharic languages.”

Another issue raised during focus group discussions is incineration is not the final disposal method. It needs additional disposal sites, lacks technology, is costly to construct a brick incinerator, lacks knowledge for health facility workers, shortage of man powers /cleaners, absence of environmental health professionals in health centres and all private clinics, and continues exposure to the staff for needle stick injury, foully smell, human scavengers, unsightly, fire hazard, and lack of water supply in the town are the major teams that FGD participants raise and forwarded the above issue as a problem to improve SHCWMP.

Focus group participants, during the discussion, raised issues that could be more comfortable managing SHCWs properly in their institution. Two of the 37 private health facilities are working in their own compound, and the remaining 35 are rented; because of this, they have difficulty constructing incinerators and ash removal pits and are not confident about investing in SHCWM systems. Staff negligence and involuntary abiding by the rules of the facilities were raised by four of the government health facilities, and it was difficult to punish those who violated the healthcare waste management rules because the health facility leaders were not giving appropriate attention to the problem.

Focus group participants forwarded recommendations on which interventions can improve the management of SHCW, and recommendations are summarised as follows:

“PPE should be available in quality and quantity for all health facility workers who have direct contact with SHCW.” “Scientific-based waste management technologies should be availed for health facilities.” “Continuous induction HCW management training should be provided to the workers. Law enforcement should be strengthened.” “Communal HCW management sites should be availed, especially for private health facilities.” “HCWM committee should be strengthened.” “Non-infectious wastes should be collected communally and transported to the municipal SHCW disposal places.” “Leaders should be knowledgeable on the SHCWM system and supervise the practice continuously.” “Patient and client should be oriented daily about HCW segregation practice.” “Regulatory bodies should supervise the health facilities before commencing and periodically between services .”

The above are the themes that FGD participants discussed and forwarded for the future improvements of SHAWMP in the study areas.

Lack of water supply in the town

Other issues raised during FGDs were health facilities’ lack of water supply. World Health Organization (2014: 89) highlights that water supply for the appropriate waste management system should be mandatory at any time in all health service delivery points.

Thirty-nine (95.12%) of the health facilities complain about the absence of water supply to improve HCW management practices and infection prevention and control practices in the facilities.

“We get water once per week, and most of the time, the water is available at night, and if we are not fetching as scheduled, we can’t get water the whole week”.

In this research, only those who have direct contact have participated in this study, and 434 (80.4%) of the respondents agree they have roles and responsibilities for appropriate solid health care waste management practice. The rest, 19.6%, do not agree with their commitment to manage health care wastes properly, even though they are responsible. Health facility workers in NEMMCSH and medium clinics know their responsibilities better than others, and their results show above the mean. 84.5%, 74.5%, 81%, 73.9% and 75% in NEMMCSH, Government health centres, medium clinics, small clinics, and surgical centres, respectively.

Establishing a policy and a legal framework, training personnel, and raising public awareness are essential elements of successful healthcare waste management. A policy can be viewed as a blueprint that drives decision-making at a political level and should mobilize government effort and resources to create the conditions to make changes in healthcare facilities. Three hundred and seventy-four (69.3%) of the respondents agree with the presence of any solid healthcare waste management policy in Ethiopia. The more knowledge above the mean (72.9%) on the presence of the policy is reported from NEMMCSH.

Self-reported level of knowledge on what to do in case of an accident revealed that 438 (81.1% CI: 77.6–84.3%) of the respondents knew what to do in case of an accident. Government health centre staff and medium clinic staff’s knowledge about what to do in case of an accident was above the mean (88.4% and 82.3%), respectively, and the rest were below the mean. The action performed after an occupational accident revealed that 56 (35.7%) of the respondents did nothing after any exposure to an accident. Out of 56 respondents who have done nothing after exposure, 47 (83.92%) of the respondents answered yes to their knowledge about what to do in case of an accident. Out of 157 respondents who have been exposed to occupational accidents, only 59 (37.6%) of the respondents performed the appropriate measures, 18 (11.5%), 9 (5.7%), 26 (16.6%), 6 (3.8%) of the respondents are taking prophylaxis, linked to the incident officer, consult the available doctors near to the department, and test the status of the patient (source of infection) respectively and the rest were not performing the scientific measures, that is only practising one of the following practices washing the affected part, squeezing the affected part to remove blood, cleaning the affected part with alcohol.

Health facility workers’ understanding of solid health care waste management practices was assessed by asking whether the current SHCWM practice needs improvement. Four hundred forty-nine (83.1%) health facility workers are unsatisfied with the current solid waste management practice at the different health facility levels, and they recommend changing it to a scientific one. 82.6%, 87.4%, 89.9%, 75%, and 81.3% of the respondents are uncomfortable or need to improve solid health care waste management practices in NEMMCSH, government health centres, medium clinics, small clinics, and surgical centres, respectively.

Lack of safety box, lack of colour-coded waste bins, lack of training, and no problems are the responses to the question problems encountered in managing SHCWMP. Two Hundred and Fifty (46.92%) and 232 (42.96%) of the respondents recommend the availability of safety boxes and training, respectively.

Four or 9.8% of the facilities have infection prevention and control (IPC) teams in the study health facilities. This finding differed from the study in Pakistan, where thirty per cent (30%) of the study hospitals had HCWM or infection control teams [ 21 ]. This study’s findings were similar to those conducted in Pakistan by Khan et al. [ 21 ], which confirmed that the teams were almost absent at the secondary and primary healthcare levels [ 20 ].

The availability of health care waste management policy report reveals that 69.3% (95% CI: 65.4–73) of the staff are aware of the presence of solid health care waste management policy in the institution. Availability of health care waste management policy was 188 (72.9%), 66 (69.5%), 53 (677.1%), 57 (62%), 10 (62.5%) in NEMMCSH, Government health centres, medium clinics, small clinics, and surgical centre respectively. Healthcare waste management policy availability was above the mean in NEMMCSH and government health centres; see Table  6 below.

Open-ended responses on the SHCWM practice of health facility workers were collected using the prepared interview guide, and the responses were analyzed using thematic analysis. All the answered questions were tallied on the paper and exported to Excel software for thematic analysis.

The study participants recommend.

“appropriate segregation practice at the point of generation” "health facility must avail all the necessary supplies that used for SHCWMP, punishment for those violating the rule of SHCWMP",

“waste management technologies should be included in solid waste management guidelines, and enforcement should be strengthened.”

The availability of written national or adopted/adapted SHCWM policies was observed at all study health facilities. Twenty eight (11.66%) of the rooms have either a poster or a written document of the national policy document. However, all staff working in the observed rooms have yet to see the inside content of the policy. The presence of the policy alone cannot bring change to SHCWMP. This finding shows that the presence of policy in the institution was reasonable compared to the study findings in Menelik II hospital in Addis Ababa, showing that HCWM regulations and any applicable facility-based policy and strategy were not found [ 22 ]. The findings of this study were less compared to the study findings in Pakistan; 41% of the health facilities had the policy document or internal rules for the HCWM [ 21 ].

Focus group participants have forwarded recommendations on which interventions can improve the management of SHCW, and recommendations are summarised as follows.

‘‘Supplies should be available in quality and quantity for all health facility workers with direct contact with SHCW. Scientific-based waste management technologies should be available for health facilities. Continues and induction health care waste management training should be provided to the workers. Law enforcement should be strengthened. Community healthcare waste management sites should be available, especially for private health facilities. HCWM committee should be strengthened. Non-infectious wastes should be collected communally and transported to the municipal SHCW disposal places. Leaders should be knowledgeable about the SHCWM system and supervise the practice continuously. Patients and clients should be oriented daily about health care waste segregation practices. Regulatory bodies should supervise the health facilities before commencing and periodically in between the service are the themes those FGD participants discussed and forward for the future improvements of SHCWMP in the study areas.’’

The availability of PPEs in different levels of health facilities shows 392 (72.6%), 212 (82.2%), 56 (58.9%), 52 (65.8%), 60 (65.2%), 12 (75%) health facility workers in NEMMCSH, government health centres, medium clinics, small clinics, and surgical centres respectively agree to the presence of personal protective equipment in their department. The availability of PPEs in this study was nearly two-fold when compared to the study findings in Myanmar, where 37.6% of the staff have PPEs [ 12 ].

The mean availability of masks, heavy-duty gloves, boots, and aprons was 71.1%, 65.4%, 38%, and 44.4% in the study health facilities. This finding shows masks are less available in the study health facilities compared to other studies. The availability of utility gloves, boots, and plastic aprons is good in this study compared to the study conducted by Banstola, D in Pokhara Sub-Metropolitan City [ 23 ].

The findings of this study show there is a poor segregation practice, and all kinds of solid wastes were collected together. This finding was similar to the study findings conducted in Addis Ababa, Ethiopia, by Debere et al. [ 24 ] and contrary to the study findings conducted in Nepal and India, which shows 50% and 65–75% of the surveyed health facilities were practising proper waste segregation systems at the point of generation without mixing general wastes with hazardous wastes respectively [ 9 , 17 ].

Ninety percent of private health facilities collect and transport SHCW generated in every service area and transport it to the disposal place by the collection container (no separate container to collect and transport the waste to the final disposal site). This finding was similar to the study findings of Debre Markos’s town [ 25 ]. At all of the facilities in the study area, SHCW was transported from the service areas to the disposal site manually by carrying the collection container, and there was no trolley for transportation. This finding was contrary to the study findings conducted in India, which show segregated waste from the generation site was being transported through the chute to the carts placed at various points on the hospital premises by skilled sanitary workers [ 17 ].

Observational findings revealed that pre-treatment of SHCW before disposal was not practised at all study health facilities. This study was contrary to the findings of Pullishery et al. [ 26 ], conducted in Mangalore, India, which depicted pre-treatment of the waste in 46% of the hospitals [ 26 ]. 95% of the facilities have no water supply for handwashing during and after solid healthcare waste generation, collection, and disposal. This finding was contrary to the study findings in Pakistan hospitals, which show all health facilities have an adequate water supply near the health care waste management sites [ 27 ].

Questionnaire data collection tools show that 129 (23.8%) of the staff needle stick injuries have occurred on health facility workers within one year of the period before the data collection. This finding was slightly smaller than the study findings of Deress et al. [ 25 ] in Debre Markos town, North East Ethiopia, where 30.9% of the workers had been exposed to needle stick injury one year prior to the study [ 25 ]. Reported and registered needle stick injuries in health facilities are less reported, and only 70 (54.2%) of the injuries are reported to the health facilities. This finding shows an underestimation of the risk and the problem, which was supported by the study conducted in Menilik II hospitals in Addis Ababa [ 22 ]. 50%, 33.4%, 48%, 52%, and 62.5% of needle stick injuries were not reported in NEMMCSH, Government health centres, medium clinics, small clinics, and surgical centres, respectively, to the health facility manager.

Nearly 1/3 (177 or 32.7%) of the staff are exposed to needle stick injuries. Needle stick injuries in health facilities are less reported, and only 73 (41.24%) of the injuries are reported to the health facilities within 12 months of the data collection. This finding is slightly higher than the study finding of Deress et al. [ 25 ] in Debere Markos, Ethiopia, in which 23.3% of the study participants had encountered needle stick/sharps injuries preceding 12 months of the data collection period [ 25 ].

Seventy-three injuries were reported to the health facility manager in the last one year, 44 of the injuries were reported by health professionals, and the rest were reported by supportive staff. These injuries were reported from 35(85.3%) health facilities; the remaining six have no report. These study findings were better than the findings of Khan et al. [ 21 ], in which one-third of the facilities had a reporting system for an incident, and almost the same percentage of the facilities had post-exposure procedures in both public and private sectors [ 21 ].

Within one year of the study period, 129 (23.88%) needle stick injuries occurred. However, needle stick injuries in health facilities are less reported, and only 70 (39.5%) of the injuries are reported to the health facilities. These findings were reasonable compared to the study findings of the southwest region of Cameroon, in which 50.9% (110/216) of all participants had at least one occupational exposure [ 28 , 29 ]. This result report shows a very high exposure to needle stick injury compared to the study findings in Brazil, which shows 6.1% of the research participants were injured [ 27 ].

The finding shows that 220 (40.8%) of the respondents were vaccinated to prevent themselves from health facility-acquired infection. One Hundred Fifty-six (70.9%) of the respondents are vaccinated in order to avoid themselves from Hep B infection. Fifty-nine (26%0.8) of the respondents were vaccinated to protect themselves from two diseases that are Hep B and COVID-19. This finding was nearly the same as the study findings of Deress et al. [ 7 ],in Ethiopia, 30.7% were vaccinated, and very low compared to the study findings of Qadir et al. [ 30 ] in Pakistan and Saha & Bhattacharjya India which is 66.67% and 66.17% respectively [ 25 , 30 , 31 ].

The incineration of solid healthcare waste technology has been accepted and adopted as an effective method in Ethiopia. These pollutants may have undesirable environmental impacts on human and animal health, such as liver failure and cancer [ 15 , 16 ]. All government health facilities use incineration to dispose of solid waste. 88.4% and 100% of the wastes are incinerated in WUNEMMCSH and government health centres, respectively. This finding contradicts the study findings in the United States of America and Malaysia, which are 49–60% and 59–60 are incinerated, respectively, and the rest are treated using other technologies [ 15 , 16 ].

All study health facilities used a brick or barrel type of incinerator. The incinerators found in the study health facilities need to meet the minimum standards of solid health care waste incineration practice. These findings were similar to the study findings of Nepal and Pakistan [ 32 ]. The health care waste treatment system in health facilities was found to be very unsystematic and unscientific, which cannot guarantee that there is no risk to the environment and public health, as well as safety for personnel involved in health care waste treatment. Most incinerators are not properly operated and maintained, resulting in poor performance.

All government health facilities use incineration to dispose of solid waste. All the generated sharp wastes are incinerated using brick or barrel incinerators, as shown in Fig.  1 above. This finding was consistent with the findings of Veilla and Samwel [ 33 ], who depicted that sharp waste generation is the same as sharps waste incinerated [ 33 ]. All brick incinerators were constructed without appropriate air inlets to facilitate combustion except in NEMMCSH, which is built at a 4-m height. These findings were similar to the findings of Tadese and Kumie at Addis Ababa [ 34 ].

Barrel and brick incinerators used in private clinic

Strengths and limitations

This is a mixed-method study; both qualitative and quantitative study design, data collection and analysis techniques were used to understand the problem better. The setting for this study was one town, which is found in the southern part of the country. It only represents some of the country’s health facilities, and it is difficult to generalize the findings to other hospitals and health centres. Another limitation of this study was that private drug stores and private pharmacies were not incorporated.

Conclusions

In the study, health facilities’ foot-operated solid waste dust bins are not available for healthcare workers and patients to dispose of the generated wastes. Health facility managers in government and private health institutions should pay more attention to the availability of colour-coded dust bins. Most containers are opened, and insects and rodents can access them anytime. Some of them are even closed (not foot-operated), leading to contamination of hands when trying to open them.

Healthcare waste management training is mandatory for appropriate healthcare waste disposal. Healthcare-associated exposure should be appropriately managed, and infection prevention and control training should be provided to all staff working in the health facilities.

Availability of data and materials

The authors declare that data for this work are available upon request to the first author.

Chartier, Y et al. Safe management of wastes from health-care activities. 2nd ed. WHO; 2014.

Tesfahun E, et al. Developing models for the prediction of hospital healthcare waste generation rate. Waste Manag Res. 2014;34(1):75–80.

Manzoor J, Sharma M. Impact of Biomedical Waste on Environment and Human Health. Environmental Claims Journal. 2019;31(4):311–34.

Article   Google Scholar  

Yves C, Jorge E, Ute P, Annette P, et al. Safe management of wastes from health-care activities. WHO 2nd ed. 2014.

OSHA. Occupational Safety and Health Administration, Guidelines for Healthcare Waste Management. 2023.

Godfrey L, Ahmed M, et al. Solid waste management in Africa: governance failure or development opportunity?. Intech open. 2019.

Deress T, Jemal M, Girma M, Adane K. Knowledge, attitude, and practice of waste handlers about medical waste management in Debre Markos town healthcare facilities, northwest Ethiopia. BMC Res Notes. 2019;12(1):146.

Article   PubMed   PubMed Central   Google Scholar  

Sahiledengle B. Self-reported healthcare waste segregation practice and its correlate among healthcare workers in hospitals of Southeast Ethiopia. BMC Health Serv Res. 2019;19(1):591.

Debalkie D, Kume A. Healthcare Waste Management: The Current Issue in Menellik II Referral Hospital, Ethiopia. Curr World Environ. 2017;12(1):42–52.

Debere MK, Gelaye KA, Alamdo AG, Trifa ZM. Assessment of the health care waste generation rates and its management system in hospitals of Addis Ababa, Ethiopia, 2011. BMC Public Health. 2013;13(28).

Creswell JW. Research design qualitative, quantitative, & mixed method approach. 4th ed. SAGE Publications, Inc.; 2014.

Win EM, Saw YM, Oo KL, Than TM, Cho SM, Kariya T, et al. Healthcare waste management at primary health centres in Mon State, Myanmar: the comparisons between hospital and non-hospital type primary health centres. Nagoya J Med Sci. 2019;81(1):81–91.

PubMed   PubMed Central   Google Scholar  

WHO. Safe management of wastes from health-care activities. 2nd ed. editor Chartier, Y et al. 2014. 

Richard B, Ben A, Kristian S. Health care without harm climate-smart health care series green paper number one. 2019.

Khadem Ghasemi M, Mohd YR. Advantages and Disadvantages of Healthcare Waste Treatment and Disposal Alternatives: Malaysian Scenario. Pol J Environ Stud. 2016;25(1):17–25.

Mohseni-Bandpei A, Majlesi M, Rafiee M, Nojavan S, Nowrouz P, Zolfagharpour H. Polycyclic aromatic hydrocarbons (PAHs) formation during the fast pyrolysis of hazardous health-care waste. Chemosphere. 2019;227:277–88.

Article   PubMed   CAS   Google Scholar  

Pandey A, Ahuja S, Madan M, Asthana AK. Bio-Medical Waste Managment in a Tertiary Care Hospital: An Overview. J Clin Diagn Res. 2016;10(11):DC01-DC3.

Doylo T, Alemayehu T, Baraki N. Knowledge and Practice of Health Workers about Healthcare Waste Management in Public Health Facilities in Eastern Ethiopia. J Community Health. 2019;44(2):284–91.

Article   PubMed   Google Scholar  

Hosny G, Samir S, Sharkawy R. An intervention significantly improve medical waste handling and management: A consequence of raising knowledge and practical skills of health care workers. Int J Health Sci.2018;12(4).

Khan EA, Sabeeh SM, Chaudhry MA, Yaqoob A, Kumar R. et al. Health care waste management in Pakistan: A situational analisis and way forward. Pak J Public Health. 2016;6(3).

Khan BA, Cheng L, Khan AA, Ahmed H. Healthcare waste management in Asian developing countries: A mini review. Waste Manag Res. 2019;37(9):863–75.

Debalkie D, Kumie A. Healthcare Waste Management: The Current Issue in Menellik II Referral Hospital. Ethiopia Current World Environment. 2017;12(1):42–52.

Banstola D, Banstola R, Nepal D, Baral P. Management of hospital solid wastes: A study in Pokhara sub metropolitan city. J Institute Med. 2017;31(1):68–74.

Debere MK, Gelaye KA, Alamdo AG, Trifa, ZM. Assessment of the HCW generation rates and its management system in hospitals of Addis Ababa, Ethiopia. BMC Public Health. 2014;13(28):1–9.

Deress T, Hassen F, Adane K, Tsegaye A. Assessment of Knowledge, Attitude, and Practice about Biomedical Waste Management and Associated Factors among the Healthcare Professionals at Debre Markos Town Healthcare Facilities. Northwest Ethiopia J Environ Public Health. 2018;2018:7672981.

PubMed   Google Scholar  

Pullishery F, Panchmal GS, Siddique S, Abraham A. Awareness, knowledge, and practices on bio-medical waste management among health care professionals in Mangalore- A cross sectional study. Integr Med. 2016;3(1):29–35.

Google Scholar  

Ream PS, Tipple AF, Salgado TA, Souza AC, Souza SM, Galdino-Junior H, et al. Hospital housekeepers: Victims of ineffective hospital waste management. Arch Environ Occup Health. 2016;71(5):273–80.

Ngwa CH, Ngoh EA, Cumber SN. Assessment of the knowledge, attitude and practice of health care workers in Fako division on post exposure prophylaxis to blood borne viruses: a hospital based cross-sectional study. Pan Afr Med J. 2018;31.

Health care waste managemnt in pakistan. a situation analysis and way forward. Pakistan Journal of Public Health. 2016;6(3):35–45.

Qadir DM, Murad DR, Faraz DN. Hospital Waste Management; Tertiary Care Hospitals. The Professional Medical Journal. 2016;23(07):802–6.

Saha A, Bhattacharjya H. Health-Care Waste Management in Public Sector of Tripura, North-East India: An Observational Study. Indian J Community Med. 2019;44(4):368–72.

Pullishery F, Panchmal G, Siddique S, Abraham A. Awareness, knowledge, and practices on bio-medical waste management among health care professionals in Mangalore- A cross sectional study. Integr Med. 2016;3(1):29–35.

Veilla EM, Samwel VM. Assessment of sharps waste management practices in a referral hospital. Afr J Environ Sci Technol. 2016;10(3):86–95.

Tadesse ML, Kumie A. Healthcare waste generation and management practice in government health centers of Addis Ababa. Ethiopia BMC Public Health. 2014;14:1221.

Download references

Acknowledgements

The authors are grateful to the health facility leaders and ethical committees of the hospitals for their permission. The authors acknowledge the cooperation of the health facility workers who participated in this study.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Wachemo University College of Medicine and public health, Hossana, Ethiopia

Yeshanew Ayele Tiruneh

Department of Public Health, University of South Africa, College of Human Science, Pretoria, South Africa

L. M. Modiba & S. M. Zuma

You can also search for this author in PubMed   Google Scholar

Contributions

Dr. Yeshanew Ayele Tiruneh is a researcher of this study; the principal investigator does all the proposal preparation, methodology, data collection, result and discussion, and manuscript writing. Professor LM Modiba and Dr. SM Zuma are supervisors for this study. They participated in the topic selection and modification to the final manuscript preparation by commenting on and correcting the study. Finally, the three authors read and approved the final version of the manuscript and agreed to submit the manuscript for publication.

Corresponding author

Correspondence to Yeshanew Ayele Tiruneh .

Ethics declarations

Ethics approval and consent to participate.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Tiruneh, Y.A., Modiba, L.M. & Zuma, S.M. Solid health care waste management practice in Ethiopia, a convergent mixed method study. BMC Health Serv Res 24 , 985 (2024). https://doi.org/10.1186/s12913-024-11444-8

Download citation

Received : 05 March 2023

Accepted : 14 August 2024

Published : 26 August 2024

DOI : https://doi.org/10.1186/s12913-024-11444-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health care waste
• Waste management
• Private health facilities

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Chapter 7. Reporting the Results of Mixed Method Evaluations

The final task the evaluator is required to perform is to summarize what the team has done, what has been learned, and how others might benefit from this project’s experience. As a rule, NSF grantees are expected to submit a final report when the evaluation has been completed. For the evaluator, this is seen as the primary reporting task, which provides the opportunity to depict in detail the rich qualitative and quantitative information obtained from the various study activities.

In addition to the contracting agency, most evaluations have other audiences as well, such as previously identified stakeholders, other policymakers, and researchers. For these audiences, whose interest may be limited to a few of the topics covered in the full report, shorter summaries, oral briefings, conference presentations, or workshops may be more appropriate. Oral briefings allow the sharing of key findings and recommendations with those decisionmakers who lack the time to carefully review a voluminous report. In addition, conference presentations and workshops can be used to focus on special themes or to tailor messages to the interests and background of a specific audience.

In preparing the final report and other products that communicate the results of the evaluation, the evaluator must consider the following questions:

This chapter deals primarily with these questions. More extensive coverage of the general topic of reporting and communicating evaluation results can be found in the earlier User-Friendly Handbook for Project Evaluation (NSF, 1993).

Ascertaining the Interests and Needs of the Audience

The diversity of audiences for which the findings are likely to be of interest is illustrated for the hypothetical project in Exhibit 19. As shown, in addition to NSF, the immediate audience for the evaluation might include top-level administrators at the major state university, staff at the Center for Educational Innovation, and the undergraduate faculty who are targeted to participate in these or similar workshops. Two other indirect audiences might be policymakers at other 4-year institutions interested in developing similar preservice programs and other researchers. Each of these potential audiences might be interested in different aspects of the evaluation's findings. Not all data collected in a mixed method evaluation will be relevant to their interests. For example:

In this example, the evaluator would risk having the results ignored by some stakeholders and underutilized by others if only a single dissemination strategy was used. Even if a single report is developed for all stakeholders (which is usually the case), it is advisable to develop a dissemination strategy that recognizes the diverse informational needs of the audience and the limited time some readers might realistically be able to devote to digesting the results of the study. Such a strategy might include (1) preparing a concise executive summary of the evaluation’s key findings (for the university's top-level administrators); (2) preparing a detailed report (for the Center for Educational Innovation and the National Science Foundation) that describes the history of the program, the range of activities offered to undergraduate faculty, and the impact of these activities on program participants and their students; and (3) conducting a series of briefings that are tailored to the interests of specific stakeholders (e.g., university administrators might be briefed on the program's tangible benefits and costs). By referring back to the worksheets that were developed in planning the evaluation (see Chapter 5), the interests of specific stakeholders can be ascertained. However, rigid adherence to the original interests expressed by stakeholders is not always the best approach. This strategy may shortchange the audience if the evaluation - as is often the case - pointed to unanticipated developments. It should also be pointed out that while the final report usually is based largely on answers to summative evaluation questions, it is useful to summarize salient results of the formative evaluation as well, where these results provide important information for project replication.

Organizing and Consolidating the Final Report

Usually, the organization of mixed method reports follows the standard format, described in detail in the earlier NSF user-friendly handbook, that consists of five major sections:

In addition to the main body of the report, a short abstract and a one-to four-page executive summary should be prepared. The latter is especially important because many people are more likely to read the executive summary than the full document. The executive summary can help focus readers on the most significant aspects of the evaluation. It is desirable to keep the methodology section short and to include a technical appendix containing detailed information about the data collection and other methodological issues. All evaluation instruments and procedures should be contained in the appendix, where they are accessible to interested readers.

Regardless of the audience for which it is written, the final report must engage the reader and stimulate attention and interest. Descriptive narrative, anecdotes, personalized observations, and vignettes make for livelier reading than a long recitation of statistical measures and indicators. One of the major virtues of the mixed method approach is the evaluator’s ability to balance narrative and numerical reporting. This can be done in many ways: for example, by alternating descriptive material (obtained through qualitative techniques) and numerical material (obtained through quantitative techniques) when describing project activities, or by using qualitative information to illustrate, personalize, or explicate a statistical finding.

But - as discussed in the earlier chapters - the main virtue of using a mixed method approach is that it enlarges the scope of the analysis. And it is important to remember that the purpose of the final report is not only to tell the story of the project, its participants, and its activities, but also to assess in what ways it succeeded or failed in achieving its goals.

In preparing the findings section, which constitutes the heart of the report, it is important to balance and integrate the descriptive and evaluative reporting section. A well-written report should provide a concise context for understanding the conditions in which results were obtained and identifying specific factors (e.g., implementation strategies) that affected the results. According to Patton (1990),

Description is thus balanced by analysis and interpretation. Endless description becomes its own muddle. The purpose of analysis is to organize description so that it is manageable. Description is balanced by analysis and leads into interpretation. An interesting and reasonable report provides sufficient description to allow the reader to understand the basis for an interpretation, and sufficient interpretation to allow the reader to understand the description.

For the hypothetical project, most questions identified for the summative evaluation in Exhibit 16 can be explored through the joint use of qualitative and quantitative data, as shown in Exhibit 20. For example, to answer some of the questions pertaining to the impact of faculty training on their students’ attitudes and behaviors, quantitative data (obtained from a student survey) are being used, together with qualitative information obtained through several techniques (classroom observations, faculty focus groups, interviews with knowledgeable informants.)

Formulating Sound Conclusions and Recommendations

In the great majority of reporting activities, the evaluator will seek to include a conclusion and recommendations section in which the findings are summarized, broader judgments are made about the strengths and weaknesses of the project and its various features, and recommendations for future, perhaps improved, replications are presented. Like the executive summary, this section is widely read and may affect policymakers' and administrators' decisions with respect to future project support.

The report writer can include in this section only a limited amount of material, and should therefore select the most salient findings. But how should saliency be defined? Should the "strongest" findings be emphasized, i.e., those satisfying accepted criteria for soundness in the quantitative and qualitative traditions? Or should the writer present more sweeping conclusions, ones which may be based in part on impressionistic and anecdotal material?

It can be seen that the evaluator often faces a dilemma. On the one hand, it is extremely important that the bulk of evaluative statements made in this section can be supported by accurate and robust data and systematic analysis. As discussed in Chapter 1, some stakeholders may seek to discredit evaluations if they are not in accord with their expectations or preferences; such critics may question the conclusions and recommendations offered by the evaluator that seem to leap beyond the documented evidence. On the other hand, it is often beneficial to capture insights that result from immersion in a project, insights not provided by sticking only to results obtained through scientific documentation. The evaluator may have developed a strong, intuitive sense of how the project really worked out, what were its best or worst features, and what benefits accrued to participants or to institutions impacted by the project (for example, schools or school systems). Thus, there may be a need to stretch the data beyond their inherent limits, or to make statements for which the only supporting data are anecdotal.

We have several suggestions for dealing with these issues:

Maintaining Confidentiality

All research involving human subjects entails possible risks for participants and usually requires informed consent on their part. To obtain this consent, researchers usually assure participants that their identity will not be revealed when the research is reported and that all information obtained through surveys, focus groups, personal interviews, and observations will be handled confidentially. Participants are assured that the purpose of the study is not to make judgments about their performance or behavior, but simply to improve knowledge about a project's effectiveness and improve future activities.

In quantitative studies, reporting procedures have been developed to minimize the risk that the actions and responses of participants can be associated with a specific individual; usually results are reported for groupings only and, as a rule, only for groupings that include a minimum number of subjects.

In studies that use qualitative methods, it may be more difficult to report all findings in ways that make it impossible to identify a participant. The number of respondents is often quite small, especially if one is looking at respondents with characteristics that are of special interest in the analysis (for example, older teachers, or teachers who hold a graduate degree). Thus, even if a finding does not name the respondent, it may be possible for someone (a colleague, an administrator) to identify a respondent who made a critical or disparaging comment in an interview.

Of course, not all persons who are interviewed in the course of an evaluation can be anonymous: the name of those persons who have a unique or high status role (the project director, a college dean, or a school superintendent) are known, and anonymity should not be promised. The issue is of importance to more vulnerable persons, usually those in subordinate positions (teachers, counselors, or students) who may experience negative consequences if their behavior and opinions become known.

It is in the interest of the evaluator to obtain informed consent from participants by assuring them that their participation is risk-free; they will be more willing to participate and will speak more openly. But in the opinion of experienced qualitative researchers, it is often impossible to fulfill promises of anonymity when qualitative methods are used:

Confidentiality and anonymity are usually promised - sometimes very superficially - in initial agreements with respondents. For example, unless the researcher explains very clearly what a fed-back case will look like, people may not realize that they will not be anonymous at all to other people within the setting who read the case (Miles and Huberman, 1994).

The evaluator may also find it difficult to balance the need to convey contextual information that will provide vivid descriptive information and the need to protect the identity of informants. But if participants have been promised anonymity, it behooves the evaluator to take every precaution so that informants cannot be linked to any of the information they provided.

In practice, the decision of how and when to attribute findings to a site or respondent is generally made on a case-by-case basis. The following example provides a range of options for revealing and disclosing the source of information received during an interview conducted for the hypothetical project:

Each of these alternatives has consequences not only for protecting respondent anonymity, but also for the value of the information that is being conveyed. The first formulation discloses the identity of the respondent and should only be used if anonymity was not promised initially, or if the respondent agrees to be identified. The last alternative, while offering the best guarantee of anonymity, is so general that it weakens the impact of the finding. Depending on the direction taken by the analysis (were there important differences by site? by type of respondent?), it appears that either the second or third alternative 2 or 3 represents the best choice.

One common practice is to summarize key findings in chapters that provide cross-site analyses of controversial issues. This alternative is "directly parallel to the procedure used in surveys, in which the only published report is about the aggregate evidence" (Yin, 1990). Contextual information about individual sites can be provided separately, e.g., in other chapters or an appendix.

Tips for Writing Good Evaluation Reports

Start early. Although we usually think about report writing as the final step in the evaluation, a good deal of the work can (and often does) take place before the data are collected. For example, a background section can often be developed using material from the original proposal. While some aspects of the methodology may deviate from the original proposal as the study progresses, most of the background information (e.g., nature of the problem, project goals) will remain the same throughout the evaluation. In addition, the evaluation study questions section can often be written using material that was developed for the evaluation design. The evaluation findings, conclusions, and recommendations generally need to wait for the end of the evaluation.

Because of the volume of written data that are collected on site, it is generally a good idea to organize study notes as soon after a site visit or interview as possible. These notes will often serve as a starting point for any individual case studies that might be included in the report. In addition, as emphasized in Chapter 4, preparing written text soon after the data collection activity will help to classify and display the data and reduce the overall volume of narrative data that will eventually need to be summarized and reported at the end of the study. Finally, preparing sections of the findings chapter during the data collection phase allows researchers to generate preliminary conclusions or identify potential trends that can be confirmed or refuted by additional data collection activities.

Make the report concise and readable. Because of the volume of material that is generally collected during mixed method evaluations, a challenging aspect of reporting is deciding what information might be omitted from the final report. As a rule, only a fraction of the tabulations prepared for survey analysis need to be displayed and discussed. Qualitative field work and data collection methods yield a large volume of narrative information, and evaluators who try to incorporate all of the qualitative data they collected into their report risk losing their audience. Conversely, by omitting too much, evaluators risk removing the context that helps readers attach meaning to any of the report's conclusions. One method for limiting the volume of information is to include only narrative that is tied to the evaluation questions. Regardless of how interesting an anecdote is, if the information does not relate to one of the evaluation questions, it probably does not belong in the report. As discussed previously, another method is to consider the likely information needs of your audience. Thinking about who is most likely to act upon the report's findings may help in the preparation of a useful and illuminating narrative (and in the discarding of anecdotes that are irrelevant to the needs of the reader).

The liberal use of qualitative information will enhance the overall tone of the report. In particular, lively quotes can highlight key points and break up the tedium of a technical summation of study findings. In addition, graphic displays and tables can be used to summarize significant trends that were uncovered during observations or interviews. Photographs are an effective tool to familiarize readers with the conditions (e.g., classroom size) within which a project is being implemented. New desktop publishing and software packages have made it easier to enhance papers and briefings with photographs, colorful graphics, and even cartoons. Quotes can be enlarged and italicized throughout the report to make important points or to personalize study findings. Many of these suggestions hold true for oral presentations as well.

Solicit feedback from project staff and respondents. It is often useful to ask the project director and other staff members to review sections of the report that quote information they have contributed in interviews, focus groups, or informal conversations. This review is useful for correcting omissions and misinterpretations and may elicit new details or insights that staff members failed to share during the data collection period. The early review may also avoid angry denials after the report becomes public, although it is no guarantee that controversy and demands for changes will not follow publication. However, the objectivity of the evaluation is best served if overall findings, conclusions and recommendations are not shared with the project staff before the draft is circulated to all stakeholders.

In general, the same approach is suggested for obtaining feedback from respondents. It is essential to inform them of the inclusion of data with which they can be identified, and to honor requests for anonymity. The extent to which other portions of the write-up should be shared with respondents will depend on the nature of the project and the respondent population, but in general it is probably best to solicit feedback following dissemination of the report to all stakeholders.

Miles, M.B., and Huberman, A.M. (1994). Qualitative Data Analysis , 2nd Ed. Newbury Park, CA: Sage.

National Science Foundation. (1993). User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering, and Technical Education . NSF 93-152. Washington, DC: NSF.

Patton, M.Q. (1990). Qualitative Evaluation and Research Method , 2nd Ed. Newbury Park, CA: Sage.

Yin, R.K. (1989). Case Study Research: Design and Method. Newbury Park, CA: Sage.

Previous Chapter | Back to Top | Next Chapter

Table of Contents

Enhancing the QUAlity and Transparency Of health Research

• Courses & events
• Librarian Network
• Search for reporting guidelines

Use your browser's Back button to return to your search results

Systematic Development of Standards for Mixed Methods Reporting in Rehabilitation Health Sciences Research

Reporting guidelines for main study types

Translations

Some reporting guidelines are also available in languages other than English. Find out more in our Translations section .

• About the Library

For information about Library scope and content, identification of reporting guidelines and inclusion/exclusion criteria please visit About the Library .

Visit our Help page for information about searching for reporting guidelines and for general information about using our website.

Library index

• What is a reporting guideline?
• Browse reporting guidelines by specialty
• Reporting guidelines under development
• Translations of reporting guidelines
• EQUATOR Network reporting guideline manual
• Reporting guidelines for animal research
• Guidance on scientific writing
• Guidance developed by editorial groups
• Research funders’ guidance on reporting requirements
• Professional medical writing support
• Research ethics, publication ethics and good practice guidelines
• Links to other resources

Guidelines for Conducting Mixed-methods Research: An Extension and Illustration

Venkatesh, V., Brown, S.A., and Sullivan, Y.W. 'Guidelines for Conducting Mixed-methods Research: An Extension and Illustration,' Journal of the AIS (17:7), 2016, 435-495. https://doi.org/10.17705/1jais.00433

77 Pages Posted: 28 Jan 2022

Viswanath Venkatesh

Virginia Polytechnic Institute and State University - Pamplin College of Business

Sue A. Brown

University of Arizona - Department of Management Information Systems

Yulia Sullivan

Binghamton University

Date Written: 2016

The objective of this paper is to extend the guidelines of Venkatesh et al. (2013) for mixed methods research by identifying and integrating variations in mixed methods research. By taking into account 14 properties of mixed methods research (e.g., purposes, research questions, epistemological assumptions), our guidelines demonstrate how researchers can flexibly identify the existing variations in mixed methods research and proceed accordingly with a study design that suits their needs. To make the guidelines actionable for various situations and issues that researchers could encounter, we develop a decision tree to map the flow and relationship among the design strategies. We also provide an in-depth illustration of one possible type of mixed methods research in information systems and discuss how to develop and validate meta-inferences as the outcomes of such a study.

Keywords: mixed methods research, meta-inferences, research design, qualitative, quantitative

Suggested Citation: Suggested Citation

Viswanath Venkatesh (Contact Author)

Virginia polytechnic institute and state university - pamplin college of business ( email ).

VA United States

HOME PAGE: http://vvenkatesh.com

University of Arizona - Department of Management Information Systems ( email )

AZ United States

Binghamton University ( email )

PO Box 6001 Binghamton, NY 13902-6000 United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, psychology research methods ejournal.

Subscribe to this fee journal for more curated articles on this topic

Sociological Research Methods eJournal

Practitioner, patient and public views on the acceptability of Mobile Stroke Units in England and Wales: a mixed methods study

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for L. Moseley
• ORCID record for G McClelland
• ORCID record for D. Wilson
• ORCID record for J. Scott
• For correspondence: [email protected]
• Info/History
• Preview PDF

Background Evidence for Mobile Stroke Units (MSUs) demonstrates that onset to treatment times for intravenous thrombolysis can be reduced and access to mechanical thrombectomy might be improved. Despite growing use of MSUs internationally, to date there have been no studies in NHS England and NHS Wales exploring the acceptability of MSUs to clinicians, patient and public representatives and other key stakeholders, which are important when considering potential feasibility and implementation.

Methods This study used a mixed methods design with a cross-sectional survey and qualitative workshops and interviews between October 2023 to May 2024. Survey data were collected from clinicians involved in emergency stroke care. Qualitative data involved clinical and non-clinical professionals involved in stroke care alongside patient and public representatives with experience of stroke. Survey data were descriptively analysed while content analysis was used on open-ended questions. Qualitative data were thematically analysed, prior to triangulation using a convergent coding matrix.

Results The study results, drawn from 25 respondents to the survey and 21 participants in qualitative workshops, found that almost all participants had positive affective attitudes to the concept of MSUs. However, several key areas of concern were identified that need to be addressed prior to implementing MSUs. These concerns included how MSUs would be staffed; whether and how telemedicine could contribute; the types of economic impacts; extent to which triage systems could accurately identify stroke patients for MSUs to attend; where the base location and geographic coverage of MSUs should be, the impact of MSUs on equitable access to stroke care, and how to improve public awareness of MSUs.

Conclusion Whilst MSUs are mostly acceptable to key stakeholders, numerous areas of concern need to be addressed prior to MSU implementation. We recommend further research to address these issues prior to implementation in the NHS.

Competing Interest Statement

The authors have declared that no competing interests exist.

Funding Statement

Author declarations.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Ethical approval was provided via Northumbria University ethics online system (reference: 4117). The study was deemed by the Health Research Authority (HRA) to not require HRA approval. All participants gave written consent prior to any data collection.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

Data cannot be shared publicly because participants did not give consent for data sharing. Data are available from Northumbria University, contact via corresponding author, for researchers who meet the criteria for access to confidential data.

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One

Subject Area

• Addiction Medicine (345)
• Allergy and Immunology (666)
• Anesthesia (181)
• Cardiovascular Medicine (2635)
• Dentistry and Oral Medicine (314)
• Dermatology (223)
• Emergency Medicine (398)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (937)
• Epidemiology (12205)
• Forensic Medicine (10)
• Gastroenterology (756)
• Genetic and Genomic Medicine (4079)
• Geriatric Medicine (387)
• Health Economics (679)
• Health Informatics (2638)
• Health Policy (1000)
• Health Systems and Quality Improvement (983)
• Hematology (362)
• HIV/AIDS (849)
• Infectious Diseases (except HIV/AIDS) (13671)
• Intensive Care and Critical Care Medicine (793)
• Medical Education (399)
• Medical Ethics (109)
• Nephrology (433)
• Neurology (3856)
• Nursing (209)
• Nutrition (573)
• Obstetrics and Gynecology (736)
• Occupational and Environmental Health (693)
• Oncology (2015)
• Ophthalmology (584)
• Orthopedics (240)
• Otolaryngology (304)
• Pain Medicine (250)
• Palliative Medicine (75)
• Pathology (472)
• Pediatrics (1113)
• Pharmacology and Therapeutics (464)
• Primary Care Research (449)
• Psychiatry and Clinical Psychology (3415)
• Public and Global Health (6509)
• Radiology and Imaging (1395)
• Rehabilitation Medicine and Physical Therapy (811)
• Respiratory Medicine (871)
• Rheumatology (402)
• Sexual and Reproductive Health (408)
• Sports Medicine (342)
• Surgery (442)
• Toxicology (52)
• Transplantation (185)
• Urology (165)

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic Development of Standards for Mixed Methods Reporting in Rehabilitation Health Sciences Research

Affiliations.

• 1 Department of Physical Therapy, Nova Southeastern University, Fort Lauderdale, Florida, USA.
• 2 Department of Physical Therapy & Human Movement Science, Sacred Heart University, Fairfield, Connecticut, USA.
• PMID: 37672215
• DOI: 10.1093/ptj/pzad084

Objective: Mixed methods research (MMR) integrates quantitative and qualitative methods throughout the research process to answer complex research questions. MMR designs align with the guiding frameworks of patient-centered care and social determinants of health by effectively examining the role of contextual factors and human experiences in influencing health and rehabilitation outcomes. Reporting standards and critical appraisal tools ensure the quality and transparency of the research process. MMR standards exist; yet, there is a need for reporting guidelines and an appraisal tool that meets field standards, is applicable across rehabilitation fields of study, and can accommodate the range of possibilities for combining research approaches and methods.

Methods: Mixed Methods Reporting in Rehabilitation & Health Sciences (MMR-RHS) was developed using a systematic consensus-building process in accordance with published guidance and was preregistered with the Enhancing the Quality and Transparency of Health Research Network. MMR-RHS evolved through a sequence of steps, including extensive literature review, expert consultation, stakeholder feedback, pilot testing, and tool refinement.

Results: MMR-RHS consists of 20 criteria that align with field standards for rigor and transparency, with an emphasis on integration throughout the research process, a key component of MMR.

Conclusions: A systematic process was utilized to develop the reporting standards and an appraisal tool for MMR in rehabilitation and health science. The tool is comprehensive, includes a set of criteria grounded in MMR literature, and is flexible for application to a range of MMR designs commonly seen in rehabilitation research.

Impact: The MMR-RHS may improve the quality and transparency of MMR by supporting investigators, authors, reviewers, and editors during project development, manuscript preparation, and critical review. The tool may assist readers in critical appraisal, knowledge translation, and application of published MMR findings. Ultimately, the MMR-RHS may help legitimize mixed methods in rehabilitation and health research, an important step toward understanding the complexities of health care, patient outcomes, and evolving societal health needs.

Keywords: Critical Appraisal; Mixed Methods Research; Reporting Standards; Research Reporting; Transparency.

© The Author(s) 2023. Published by Oxford University Press on behalf of the American Physical Therapy Association. All rights reserved. For permissions, please e-mail: [email protected].

PubMed Disclaimer

Similar articles

• Promoting and supporting self-management for adults living in the community with physical chronic illness: A systematic review of the effectiveness and meaningfulness of the patient-practitioner encounter. Rees S, Williams A. Rees S, et al. JBI Libr Syst Rev. 2009;7(13):492-582. doi: 10.11124/01938924-200907130-00001. JBI Libr Syst Rev. 2009. PMID: 27819974
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Standards for reporting qualitative research: a synthesis of recommendations. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. O'Brien BC, et al. Acad Med. 2014 Sep;89(9):1245-51. doi: 10.1097/ACM.0000000000000388. Acad Med. 2014. PMID: 24979285
• Health professionals' experience of teamwork education in acute hospital settings: a systematic review of qualitative literature. Eddy K, Jordan Z, Stephenson M. Eddy K, et al. JBI Database System Rev Implement Rep. 2016 Apr;14(4):96-137. doi: 10.11124/JBISRIR-2016-1843. JBI Database System Rev Implement Rep. 2016. PMID: 27532314 Review.
• Guidelines for Reporting Outcomes in Trial Protocols: The SPIRIT-Outcomes 2022 Extension. Butcher NJ, Monsour A, Mew EJ, Chan AW, Moher D, Mayo-Wilson E, Terwee CB, Chee-A-Tow A, Baba A, Gavin F, Grimshaw JM, Kelly LE, Saeed L, Thabane L, Askie L, Smith M, Farid-Kapadia M, Williamson PR, Szatmari P, Tugwell P, Golub RM, Monga S, Vohra S, Marlin S, Ungar WJ, Offringa M. Butcher NJ, et al. JAMA. 2022 Dec 20;328(23):2345-2356. doi: 10.1001/jama.2022.21243. JAMA. 2022. PMID: 36512367 Review.
• Analysis of Knowledge and Satisfaction in Virtual Clinical Simulation among Nursing Students: A Mixed Study. Medel D, Reguant M, Cemeli T, Jiménez Herrera M, Campoy C, Bonet A, Sanromà-Ortíz M, Roca J. Medel D, et al. Nurs Rep. 2024 Apr 27;14(2):1067-1078. doi: 10.3390/nursrep14020081. Nurs Rep. 2024. PMID: 38804414 Free PMC article.

Publication types

• Search in MeSH

Related information

Linkout - more resources, full text sources.

• Ovid Technologies, Inc.
• Silverchair Information Systems

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

QUAL Friendly Mixed Methods Research

• Conducting Research

Qualitative Research

September 2022

• Slides (PDF, 1MB)
• Related resources (PDF, 16KB)
• FAQs (PDF, 195KB)

This content is disabled due to your privacy settings. To re-enable, please adjust your cookie preferences.

This webinar reviews four different strategies for integrating qualitative and quantitative data or results that invite a more instrumental role for a qualitative inquiry in contributing analytical insight.

This program does not offer CE credit.

Elizabeth G. Creamer, EdD

Professor emerita, Virginia Tech.

More in this series

Presents a handful of key strategies to maintain our mental well-being when trying to help others do the same.

October 2022 On Demand Webinar

Emphasizes the basics of classic grounded theory and shows how the original tenets of the method guide the procedures.

Describes how qualitative evaluation can make a vital contribution to every stage of developing and optimizing an intervention.

Provides practical guidance to help researchers carry out IPA studies which take advantage of the strengths and potential for flexibility within the approach.

September 2022 On Demand Webinar

Journal Article Reporting Standards (JARS)

APA Style Journal Article Reporting Standards offer guidance on what information should be included in all manuscript sections for quantitative, qualitative, and mixed methods research and include how to best discuss race, ethnicity, and culture.

Introducing Journal Article Reporting Standards for Race, Ethnicity, and Culture (JARS–REC)

JARS–REC were created to develop best practices related to the manner in which race, ethnicity, and culture are discussed within scientific manuscripts in psychological science.

Quantitative research

Use JARS–Quant when you collect your study data in numerical form or report them through statistical analyses.

Qualitative research

Use JARS–Qual when you collect your study data in the form of natural language and expression.

Mixed methods research

Use JARS–Mixed when your study combines both quantitative and qualitative methods.

Race, ethnicity, culture

Use JARS–REC for all studies for guidance on how to discuss race, ethnicity, and culture.

What are APA Style JARS?

APA Style Journal Article Reporting Standards (APA Style Jars ) are a set of standards designed for journal authors, reviewers, and editors to enhance scientific rigor in peer-reviewed journal articles. Educators and students can use APA Style JARS as teaching and learning tools for conducting high quality research and determining what information to report in scholarly papers.

The standards include information on what should be included in all manuscript sections for:

• Quantitative research ( Jars –Quant)
• Qualitative research ( Jars –Qual)
• Mixed methods research ( Jars –Mixed)

Additionally, the APA Style Journal Article Reporting Standards for Race, Ethnicity, and Culture ( Jars – Rec ) provide guidance on how to discuss race, ethnicity, and culture in scientific manuscripts. Jars – Rec should be applied to all research, whether it is quantitative, qualitative, or mixed methods.

• Race, Ethnicity, and Culture ( Jars – Rec )

Using these standards will make your research clearer and more accurate as well as more transparent for readers. For quantitative research, using the standards will increase the reproducibility of science. For qualitative research, using the standards will increase the methodological integrity of research.

Jars –Quant should be used in research where findings are reported numerically (quantitative research). Jars –Qual should be used in research where findings are reported using nonnumerical descriptive data (qualitative research). Jars –Mixed should be applied to research that includes both quantitative and qualitative research (mixed methods research). JARS–REC should be applied to all research, whether it is quantitative, qualitative, or mixed methods.

For more information on APA Style JARS:

• Read Editorial: Journal Article Reporting Standards
• View an infographic (PDF, 453KB) to learn about the benefits of JARS and how they are relevant to you
• Listen to a podcast with Drs. Harris Cooper and David Frost discussing JARS and implications for research in psychology

Many aspects of research methodology warrant a close look, and journal editors can promote better methods if we encourage authors to take responsibility to report their work in clear, understandable ways. —Nelson Cowan, Editor, Journal of Experimental Psychology: General

Read more testimonials

Watch a video about JARS

This content is disabled due to your privacy settings. To re-enable, please adjust your cookie preferences.

This video describes and discusses the updated APA Style Journal Article Reporting Standards.

Related products

Reporting Qualitative Research in Psychology

Journal article reporting standards for qualitative research

Reporting Quantitative Research in Psychology

Journal article reporting standards for quantitative research

Publication Manual, 7th Edition

The official source for writing papers and creating references in seventh edition APA Style

Jars resources

• History of APA’s journal article reporting standards
• APA Style JARS supplemental glossary
• Supplemental resource on the ethic of transparency in JARS
• Frequently asked questions
• JARS-Quant Decision Flowchart (PDF, 98KB)
• JARS-Quant Participant Flowchart (PDF, 98KB)

Jars articles

• Jars –Quant article
• Jars –Qual / Mixed article
• Jars – rec executive summary

Questions / feedback

Email an APA Style Expert if you have questions, feedback, or suggestions for modules to be included in future JARS updates.

APA resources

• APA Databases and Electronic Resources
• APA Journals
• Journal Author Resource Center
• Education and Career
• Psychological Science
• Open Science at APA
• How to Review a Manuscript

From the APA Style blog

Introducing APA Style Journal Article Reporting Standards for Race, Ethnicity, and Culture

These standards are for all authors, reviewers, and editors seeking to improve manuscript quality by encouraging more racially and ethnically conscious and culturally responsive journal reporting standards for empirical studies in psychological science.

APA Style JARS for high school students

In this post, we provide an overview of APA Style JARS and resources that can be shared with high school students who want to learn more about effective communication in scholarly research.

Happy 2022, APA Stylers!

This blog post is dedicated to our awesome APA Style users. You can use the many resources on our website to help you master APA Style and improve your scholarly writing.

APA Style JARS on the EQUATOR Network

The APA Style Journal Article Reporting Standards (APA Style JARS) have been added to the EQUATOR Network. The network aims to promote accuracy and quality in reporting of research.

APA Style JARS: Resources for instructors and students

APA Style Journal Article Reporting Standards (APA Style JARS) are a set of guidelines for papers reporting quantitative, qualitative, and mixed methods research that can be used by instructors, students, and all others reading and writing research papers.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Health Serv Res
• v.47(2); 2012 Apr

Methodological Reporting in Qualitative, Quantitative, and Mixed Methods Health Services Research Articles

Jennifer p wisdom.

Psychiatry Department, Columbia University New York State Psychiatric Institute, 1051 Riverside Drive Box 100, New York, NY 10032

Mary A Cavaleri

Psychiatry Department, Columbia University New York State Psychiatric Institute, New York, NY

Anthony J Onwuegbuzie

Department of Educational Leadership Counseling at Sam Houston State University, Huntsville, TX

Carla A Green

Kaiser Permanente Northwest Center for Health Research, Portland, OR

Associated Data

Methodologically sound mixed methods research can improve our understanding of health services by providing a more comprehensive picture of health services than either method can alone. This study describes the frequency of mixed methods in published health services research and compares the presence of methodological components indicative of rigorous approaches across mixed methods, qualitative, and quantitative articles.

Data Sources

All empirical articles ( n = 1,651) published between 2003 and 2007 from four top-ranked health services journals.

Study Design

All mixed methods articles ( n = 47) and random samples of qualitative and quantitative articles were evaluated to identify reporting of key components indicating rigor for each method, based on accepted standards for evaluating the quality of research reports (e.g., use of p -values in quantitative reports, description of context in qualitative reports, and integration in mixed method reports). We used chi-square tests to evaluate differences between article types for each component.

Principal Findings

Mixed methods articles comprised 2.85 percent ( n = 47) of empirical articles, quantitative articles 90.98 percent ( n = 1,502), and qualitative articles 6.18 percent ( n = 102). There was a statistically significant difference (χ 2 (1) = 12.20, p = .0005, Cramer's V = 0.09, odds ratio = 1.49 [95% confidence interval = 1,27, 1.74]) in the proportion of quantitative methodological components present in mixed methods compared to quantitative papers (21.94 versus 47.07 percent, respectively) but no statistically significant difference (χ 2 (1) = 0.02, p = .89, Cramer's V = 0.01) in the proportion of qualitative methodological components in mixed methods compared to qualitative papers (21.34 versus 25.47 percent, respectively).

Few published health services research articles use mixed methods. The frequency of key methodological components is variable. Suggestions are provided to increase the transparency of mixed methods studies and the presence of key methodological components in published reports.

As the health services research field continues to evolve, so too does its methods. Mixed methods research capitalizes on the strengths of both qualitative and quantitative methodologies by combining approaches in a single research study to increase the breadth and depth of understanding ( Johnson, Onwuegbuzie, and Turner 2007 ). Mixed methods can be a better approach to research than either quantitative-only or qualitative-only methods when a single data source is not sufficient to understand the topic, when results need additional explanation, exploratory findings need to be generalized, or when the complexity of research objectives are best addressed with multiple phases or types of data ( Brannen 1992 ; Creswell and Plano Clark 2011 ). Rigorous mixed methods approaches require that individual components (qualitative or quantitative) adhere to their respective established standards ( Curry, Nembhard, and Bradley 2009 ; Creswell and Plano Clark 2011 ). Despite recent guidelines on frameworks for conducting mixed methods research (e.g., Curry, Nembhard, and Bradley 2009 ; Creswell and Plano Clark 2011 ), a critical challenge has been ensuring that reports from mixed methods studies transparently discuss the methodological components integral to the conduct of the studies. Health services researchers and reviewers need clear guidelines regarding research methodology, including methodological components that should be expected in mixed methods papers to indicate that they are sufficiently rigorous.

Mixed Methods in Health Services Research

Health services research is the study of how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and ultimately, health and well-being ( Lohr and Steinwachs 2002 ). As a result of the breadth of topics addressed, health services research draws upon methods and concepts from many fields, including medicine, epidemiological and economic studies, and the evaluation of services and interventions ( Field, Tranquada, and Feasley 1995 ). Health services researchers increasingly work in interdisciplinary partnerships (e.g., Aboelela et al. 2007 ) and use innovative methods, including mixed methods, to more fully understand health services phenomena. Mixed methods approaches are also consistent with suggestions to extend scientific and contextual health knowledge beyond randomized trials ( Berwick 2005 ).

Mixed methods research capitalizes on the strengths of both qualitative and quantitative methodology by combining both components in a single research study to increase breadth and depth of understanding ( Johnson, Onwuegbuzie, and Turner 2007 ). Qualitative and quantitative methods can be integrated for different purposes to provide a more comprehensive picture of health services than either method can alone. Mixed methods are appropriate in the following situations: (1) when researchers would like to converge different methods or use one method to corroborate the findings from another about a single phenomenon (triangulation); (2) when researchers would like to use one method to elaborate, illustrate, enhance, or clarify the results from another method (complementarity); (3) when researchers would like to use results from one method to inform another method, such as in creating a measure (development); (4) when researchers would like to use one method to discover paradoxes and contradictions in findings from another method that can suggest reframing research questions (initiation); and (5) when researchers seek to expand the breadth and depth of the study by using different methods for different research components (expansion) ( Greene, Caracelli, and Graham 1989 ). Bryman (2006 ) modified and expanded this list to add that mixed methods can also be useful in obtaining diversity of views, illustrating concepts, and developing instruments.

Quantitative and qualitative research can be distinguished by the philosophical assumptions brought to the study (e.g., deductive versus inductive), the types of research strategies (e.g., experiments versus case studies), and the specific research methods used in the study (e.g., structured survey versus observation) ( Creswell 2008 ). Qualitative health services research, for example, is a method in which the researcher collects textual material derived from speech or observation and attempts to understand the phenomenon of interest in terms of the meanings people bring to them ( Denzin and Lincoln 1994 ; Shortell 1999 ; Giacomini and Cook for the Evidence-Based Medicine Working Group 2000 ; Malterud 2001 ; Bradley, Curry, and Devers 2007 ). Certain characteristics are typical of qualitative research, including a naturalistic setting (as opposed to a laboratory), a focus on participants’ perspectives and their meaning, the outcome as a process rather than a product, and data collected as words or images ( Padgett 2008 ).

Guidelines for Conducting Mixed Methods Research

The National Institutes of Health noted the need for rigor in combining qualitative and quantitative methods to study complex health issues in their recent publication, Best Practices for Mixed Methods in Health Sciences ( Creswell, Klassen, Plano Clark, and Smith for the Office of Behavioral and Social Sciences Research 2011 ). There are several frameworks to guide the rigorous conduct and evaluation of mixed methods research ( Collins, Onwuegbuzie, and Sutton 2006 ; Curry, Nembhard, and Bradley 2009 ; Tashakkori and Teddlie 2010 ; Creswell and Plano Clark 2011 ). Collectively, these frameworks recommend that the conduct of mixed method studies—and reports of mixed method research, including peer-reviewed publication—demonstrates explicit rationales for all decisions regarding study design, including the purpose of including both qualitative and quantitative methods. They specifically advise that each component (qualitative or quantitative) should be conducted with a level of rigor in accordance with established principles in its field, and that researchers be transparent in methodological reporting. For example, sampling design should be specified as identical, parallel, nested, or mixed ( Onwuegbuzie and Collins 2007 ); the level of mixing methods (fully versus partially) should be described, as should time orientation (sequential or concurrent components of research) and emphasis (equal importance of methodological approaches or one more dominant) ( Leech and Onwuegbuzie 2009 ).

Conducting and evaluating mixed methods research have unique methodological challenges, particularly related to rigor. Quantitative studies typically rely on quality criteria such as internal validity, generalizability, and reliability ( Campbell 1957 ; Campbell and Stanley 1963 ; Messick 1989 , 1995 ; Onwuegbuzie and Daniel 2002 , 2004 ; Onwuegbuzie 2003 ), whereas qualitative studies have roughly comparable quality criteria of credibility, transferability, and dependability ( Lincoln and Guba 1985 ; Guba and Lincoln 1989 ; Miles and Huberman 1994 ; Maxwell 2005 ; Pope and Mays 2006 ). For example, questions asked when evaluating a qualitative study might include the following: “Were participants relevant to the research question and was their selection well reasoned?” and “Was the data collection comprehensive enough to support rich and robust descriptions of the observed events?” ( Giacomini and Cook for the Evidence-Based Medicine Working Group 2000 ). In addition to determining whether methodological approaches unique to qualitative or quantitative research were employed, an evaluation of a mixed methods study should assess aspects unique to mixed methods, such as how multiple components are integrated and how consistency and discrepancy between findings from each method are managed ( Sale and Brazil 2004 ; O'Cathain, Murphy, and Nicholl 2007 ). Qualitative, quantitative, and mixed methodologists agree that study procedures should be reported transparently, including sufficient detail to allow the reader to make inferences about study quality ( Lincoln and Guba 1985 ; Giacomini and Cook for the Evidence-based Medicine Working Group 2000 ; O'Cathain, Murphy, and Nicholl 2007 ; Armstrong et al. 2008 ; Creswell 2008 ; Curry, Nembhard, and Bradley 2009 ; Leech et al. 2009 ; Teddlie and Tashakkori 2009 ).

Several researchers have proposed specific techniques to assess the overall methodology of mixed methods research and assess the methodological components of the qualitative, quantitative, and mixed portions of the studies (e.g., Pluye et al. 2009 ; O'Cathain 2010 ; Tashakkori and Teddlie 2010 ; Creswell and Plano Clark 2011 ; Leech, Onwuegbuzie, and Combs 2011 ). For example, O'Cathain (2010 ) assessed quality of mixed methods research by evaluating transparency and clarity in reporting planning, design, data, interpretive rigor, inference transferability, reporting quality, synthesizability, and utility. Others have suggested alternative methods for assessing quality, but criteria often are not elucidated or are vague. Further, those frameworks typically address quality of the study design as opposed to the characteristics provided in the published article . By contrast, Sale and Brazil (2004 ) proposed a structured framework for the evaluation of mixed methods publications by identifying key methodological components that should be included for both qualitative and quantitative portions of studies. Despite these advances, we found few published accounts of the rigor of published mixed methods research. Our article has three specific research questions: (1) How has the frequency of mixed methods studies published in health services journals changed over time? (2) How are mixed methods articles being used to elucidate health services? and (3) To what extent do mixed methods reports differ in methodological content compared to qualitative-only or quantitative-only articles?

This systematic review assessed the frequency of mixed methods publications in top health services research journals and compared the frequency of key methodological components in qualitative, quantitative, and mixed method studies. We first reviewed articles in health services research journals to determine the prevalence of mixed methods designs and the presence of key methodological components. Then, we conducted statistical analyses of trends over time in the frequency of mixed methods articles and in the presence of key methodological components of those articles. Because this was an analysis of published data, no ethical oversight was required.

Identification of Mixed Methods Articles

We examined four journals: Health Affairs, Health Services Research , Medical Care , and Milbank Quarterly , which had 5-year impact factors of 2.94–4.71. Journals were selected by reviewing the Institute for Scientific Information (2007 ) rankings for the top 10 journals in health care sciences and services. Of these 10, we included all journals that focused generally on health services research and excluded journals with narrower foci ( Value in Health , Journal of Health Economics , Journal of Pain and Symptom Management , Statistical Methods in Medical Research , Quality and Safety in Health Care , and Quality of Life Research ). Although 2001 marked a turning point in the proliferation of mixed methods studies published in major electronic bibliographic databases such as PubMED ( Collins, Onwuegbuzie, and Jiao 2007 ), we chose to examine articles from 2003 to 2007 because 2003 marks publication of the first edition of Tashakkori and Teddlie's landmark Handbook of Mixed Methods in Social and Behavioral Research , which provided the first comprehensive collection of mixed method theory, methodology, and application. Five years represents a sufficient period of time to examine trends of published articles following the publication of a landmark methodological work.

We reviewed empirical articles to determine whether each represented a quantitative, qualitative, or mixed methods study. This entailed using all the information presented in the abstract and the body of the article to identify the research design either as stated or implied by the author(s). We excluded nonempirical articles (book reviews, literature reviews, commentaries and opinion articles, letters to the editor, policy statements) and articles from a special issue of Milbank Quarterly (Volume 83, Number 4) that included only articles published between 1932 and 1998.

We classified articles as quantitative if they included (1) a primary goal of testing theories or hypotheses about relationships between/among variables, or (2) quantitative data and methodology, such as hierarchical linear modeling, multiple regression, or Markov modeling. We classified articles as qualitative if they included either (1) a primary goal of exploring or understanding the meaning ascribed to a specific phenomenon or experience, or (2) qualitative data such as observations, unstructured or semi-structured interviews, or focus group interviews or methodologies such as thematic analysis. Although more complex definitions of mixed method studies exist (e.g., Johnson, Onwuegbuzie, and Turner 2007 ; Creswell and Plano Clark 2011 ), we classified articles as mixed methods if they integrated or combined both quantitative and qualitative methods in a single study ( Sale and Brazil 2004 ). This definition reflects the general definitions of mixed methods and the lack of consensus on a specific definition across all multidisciplinary mixed methods researchers.

We used spreadsheets to track classifications, with cells containing articles’ abstracts and our field notes. Two authors read and classified articles in batches of 50 according to type, conferring as needed until agreement was achieved ( n = 300 articles); the remaining articles ( n = 1,351) were each coded by one author. For the few articles for which methodology was ambiguous ( n = 58, 3.5 percent of all empirical articles), classification was resolved in consultation with a third author. Similar methods have been used in other evaluations of mixed methods articles ( Powell et al. 2008 ).

Assessments of Articles

We identified all mixed methods articles ( n = 47) and equal random samples ( n = 47) of quantitative articles (from 1,502 articles) and qualitative articles (from 102 articles) (total n = 141) in the four journals. Random samples of qualitative and quantitative articles were selected using a random number generator and did not adjust for journal or year. We assessed the frequency of key methodological components reported across articles, then compared rates by article type. The methodological components we focused on were drawn from two conceptual frameworks. The first included Sale and Brazil's (2004 ) criteria: (1) internal validity for quantitative findings and credibility for qualitative findings, (2) external validity for quantitative findings and transferability or fittingness for qualitative findings, (3) reliability for quantitative findings and dependability for qualitative findings, and (4) objectivity for quantitative findings and confirmability for qualitative findings (specific criteria are listed in Table 3 ). The second was O'Cathain's transparency criteria for mixed methods studies ( O'Cathain, Murphy, and Nicholl 2007 ; O'Cathain 2010 ), which specify that mixed methods studies should state the (1) priority of methods (primarily quantitative, primarily qualitative, or equal priority), (2) purpose of mixing methods (e.g., triangulation, complementarity, initiation, development, or expansion), (3) sequence of methods (qualitative first, quantitative first, or simultaneous), and (4) stage of integration of both types of data (e.g., data collection, analysis, interpretation). We assessed four additional components of mixed methods studies: (1) whether qualitative and quantitative components were integrated, (2) whether limitations of design were detailed, (3) whether areas of consistency between qualitative and quantitative components were elucidated, and (4) whether areas of inconsistency between components were described.

Key Methodological Components in Mixed Methods, Quantitative, and Qualitative Health Services Research Articles

We assessed components using categories of 0 (not described), 1 (described), or not applicable (e.g., for criteria referencing control groups in a study that had none, or ethical review for a study with no human subjects data) ( O'Cathain, Murphy, and Nicholl 2007 ). We identified only whether the study contained or did not contain each methodological component and did not attempt to assess quality or appropriateness of each component within the context of the study. For example, we assessed whether the publication stated that missing data were addressed but not whether the methods to address missing data were the best methods for that particular research design. Similar to initial article classification, two authors read and coded articles to assess presence/absence of each criterion, with any ambiguity resolved in consultation with a third author.

Quantitative Analyses of Trends and Rigor

Once all articles were coded, we conducted a statistical analysis to determine whether there were trends over time in the prevalence of mixed methods articles. To assess this, we used linear regression to test the hypothesis that there would be an increase in the prevalence of the number of mixed methods articles over time. We also conducted chi-square tests to assess differences between mixed methods, qualitative, and quantitative articles on both quantitative and qualitative criteria. We tested whether each criterion was present in the same proportion of quantitative studies as in the quantitative portion of the mixed methods studies and in the same proportion of qualitative studies as in the qualitative portion of the mixed methods studies.

In general, coders could easily categorize the type of study. Challenges arose when transparency about methods was inadequate ( N = 58, 3.5 percent of all empirical articles). For example, some papers indicated that data from interviews were included but did not provide details about who was interviewed, what was asked in the interviews, how the interview data were analyzed, or how the interview data were integrated into the overall study.

Research Question 1: How has the frequency of mixed methods studies published in health services journals changed over time?

Table 1 presents a summary of the types of articles published in four major health services research journals from 2003 through 2007. Only 2.85 percent ( n = 47) of empirical articles were mixed methods studies; 6.18 percent ( n = 102) of empirical studies represented qualitative research. Quantitative research represented 90.98 percent ( n = 1,502) of empirical articles. The journal containing the highest proportion of empirical studies employing a mixed methods design was Milbank Quarterly (8.33 percent), followed by Health Affairs (6.91 percent), Health Services Research (4.03 percent), and Medical Care (0.78 percent). Chi-square test showed a significant difference in these proportions (χ 2 = 34.67, df = 3, p < .0001).

Type and Design of Empirical Articles Published in Health Services Research Journals from 2003 to 2007, Data Presented by Journal

Note . Mixed, mixed method articles; Qual, qualitative articles; Quant, quantitative articles.

To detect temporal trends in the frequency of mixed methods research in the health services literature, articles were collapsed across journal and examined by publication year. Table 2 presents the frequency of article type for each of the 5 years. All journals combined published an average of 10.8 mixed method articles per year, or 3.27 percent of empirical articles annually. A quadratic trend was seen across the 5 years ( R 2 = 0.65), indicating a slight increase in mixed method articles in the first 2 years and then a decrease for the remaining years.

Type and Design of Empirical Articles Published in Four Health Services Research Journals from 2003 to 2007, Data Presented by Year

Research Question 2: How are mixed methods articles being used to elucidate health services research?

Mixed methods articles were categorized into four overlapping categories: Articles on organizational and individual decision making processes ( n = 18 studies) combined qualitative interviews with quantitative administrative data analyses to assess decision making about processes or impediments to processes. Examples include a study of formulary adoption decisions ( Dandrove, Hughes, and Shanley 2003 ) and states’ decisions to reduce Medicaid and other public program funding ( Hoadley, Cunningham, and McHugh 2004 ).

Sixteen articles described outcomes or effects of policies or initiatives by combining administrative health record or performance data with interviews of health administrators, providers, or executives. Examples include papers describing outcomes of pay-for-performance changes to Medicaid ( Felt-Lisk, Gimm, and Peterson 2007 ; Rosenthal et al. 2007 ) and hospital patient safety initiatives ( Devers, Pham, and Liu 2004 ).

Thirteen measurement development articles employed mixed methods to create measurement tools to assess, for example, caregiver burden ( Cousineau et al. 2003 ), patient activation ( Hibbard et al. 2004 ), and the development of a Healthcare Effectiveness Data and Information Set (HEDIS) smoking measure ( Pbert et al. 2003 ). These studies typically examined qualitative data from individual or focus group interviews first to inform creation and testing of a survey.

Articles on experiences and perceptions were the least common category ( n = 8), typically combining surveys and interviews. These included family physicians’ perceptions of the effect of medication samples on their prescribing practices ( Hall, Tett, and Nissen 2006 ); caregivers’ experiences of the termination of home health care for stroke patients ( Levine et al. 2006 ); and consumer enrollment experiences in the Cash and Counseling program ( Schore, Foster, and Phillips 2007 ).

Only five mixed methods articles (10.64 percent) of the total mixed methods sample used the terms “mixed method” or “multimethod” in the abstract or text, although four articles (8.51 percent) referred to “qualitative and quantitative” data.

Research Question 3: Do mixed methods articles report qualitative and quantitative methodology differently than methodology is reported in qualitative-only or quantitative-only articles?

Table 3 presents a summary of the frequency of key methodological components present in quantitative articles, qualitative articles, and mixed methods articles (each n = 47). For quantitative methodological components (32 items), mixed methods articles (M = 7.02 [21.94 percent], SD = 6.24) averaged statistically significantly fewer ( t (92) = −4.50, p < .00001, Cohen's d effect size = 0.93) components than did quantitative articles (M = 15.06 [47.07 percent], SD = 10.53). For qualitative methodological components (35 items), mixed methods articles (M = 7.17 [21.34 percent], SD = 6.36) did not average a statistically significantly different proportion of components ( t (92) = −1.10, p = .14, d = 0.23) than did qualitative articles (M = 8.91 [25.47 percent], SD = 8.83). No article met all criteria, and no criterion was met by all articles. For comparative analyses at a statistical significance level of α = 0.05, power to detect a medium difference (Cohen's h = 0.50) and a large difference (Cohen's h = 0.80) was 78 and 99 percent, respectively.

Of quantitative components, mixed methods studies were most likely to describe sources of data and data collection instruments (61.70 percent of studies), state the purpose/objective of the paper (59.57 percent), state the source of subjects (58.70 percent), and define/describe the study population (51.06 percent). Most mixed methods studies did not include control and intervention groups, which excluded related criteria. Quantitative studies tended to contain more key methodological components, with more than 90 percent of studies defining outcome measures (93.48 percent), defining/describing study population (91.49 percent), describing statistical procedures (95.74 percent), and stating hypotheses (97.87 percent). Quantitative studies were more likely than the quantitative portion of mixed methods studies to describe study characteristics (e.g., study design, subject recruitment), identify or control for confounding variables, provide probability values or confidence intervals, state hypotheses, or acknowledge both statistical and clinical significance (see Table 3 ).

For qualitative methodological components, mixed methods studies were most likely to state the purpose/objective of the paper (72.34 percent), triangulate qualitative sources (e.g., use both individual and focus group interviews; 53.19 percent), and describe data-gathering procedures (53.19 percent). More than 50 percent of qualitative studies triangulated qualitative sources (57.45 percent), stated the purpose/objective of the paper (57.45 percent), and described the study setting (80.43 percent), how the setting was selected (63.04 percent), the participants (55.56 percent), and data-gathering procedures (76.60 percent). Qualitative studies were more likely than the qualitative portions of the mixed methods studies to describe the study setting, justify the sampling strategy, participants, and data-gathering procedures.

For criteria regarding method integration, few authors justified the use of mixed methods or clearly described the priority, purpose, and sequence of methods, and the stage of integration. Most articles, however, integrated qualitative and quantitative components (85.11 percent); examination of articles indicated components were most frequently integrated in the interpretation phase. Across all studies, few articles stated that informed consent was obtained, ethical review was undertaken, or that subjects’ confidentiality was protected.

Previous reports indicate mixed methods articles comprised <1 percent of empirical health articles examined in 2000 ( McKibbon and Gadd 2004 ). Since then, however, the National Institutes of Health has increased funding for mixed methods research, with the proportion of funded research projects up to 5 percent of studies in some institutes ( Plano Clark 2010 ). In the United Kingdom, the proportion of funded research that uses mixed methods is at 17 percent and continuing to increase ( O'Cathain, Murphy, and Nicholl 2007 ). We found that the use of mixed methods in articles published in top health services research journals was generally consistent between 2003 and 2007 at approximately 3 percent of all empirical articles, lower than would be expected given the complexity and depth of health services research questions for which mixed methods would be appropriate. The presence of key methodological components was variable across type of article, but the quantitative portion of mixed methods articles included consistently fewer methodological components than quantitative-only studies and the qualitative portion of mixed methods articles included about the same proportion of methodological components as qualitative-only articles. Mixed methods articles also generally did not address the priority, purpose, and sequence of methods or the integration of methods as suggested by experts in mixed methods (e.g., Creswell and Tashakkori 2008 ; O'Cathain 2010 ; Creswell and Plano Clark 2011 ).

Key methodological components that cut across qualitative and quantitative methodologies were often missing from mixed methods publications. Descriptions of sample selection and sampling procedures, the study context, and data-gathering procedures are essential aspects of interpreting study findings, and mixed methods studies should not be exempt from these basic research requirements. Many mixed methods studies did not include the level of detail that would likely be required for a qualitative or quantitative paper to be accepted in these high-ranking journals. Further, the studies appeared not to follow available guidance on the structure and components of mixed methods studies that discuss basic quality criteria, data collection strategies, methods of data analysis, procedures for integration of methods, processes of making inferences from text, and recommendations for adequate reporting of results (e.g., Giacomini and Cook for the Evidence-based Medicine Working Group 2000 ; Curry, Nembhard, and Bradley 2009 ; O'Cathain 2010 ; Tashakkori and Teddlie 2010 ; Creswell and Plano Clark 2011 ). In some ways this finding is not surprising because guidance on mixed methods standards is still emerging. We expect that the National Institutes of Health publication, Best Practices for Mixed Methods in Health Sciences (Creswell, Klassen, Plano Clark, and Smith for the Office of Behavioral and Social Science Research) will lead to increased standardization of mixed methods approaches.

Although they reported more key methodological components on average than the mixed methods articles, quantitative articles in this analysis had some surprising gaps as well, including low reporting of power analyses, how missing data were addressed, and descriptions of control/comparison groups. It should be noted, however, that quantitative articles with large sample sizes do not necessarily need power analyses. With regard to single-method qualitative articles, low proportions described the study context, coding techniques, or data analysis. Few articles with human subjects involvement included statements that the research was conducted with ethical oversight, promised confidentiality, or obtained consent. These findings suggest that the issue of poor transparency in reporting methodology is not limited to mixed methods studies.

Recommendations for Mixed Methods Reporting

The methodological components reported here are not optimal indicators of the quality of mixed methods publications; an article could conceivably have all of these components and yet still be a poor research study. These components are, however, a useful starting point for a systematic evaluation of the rigor of qualitative and quantitative portions of mixed methods studies. Some journals require inclusion of other criteria (e.g., Consolidated Standards of Reporting Trials 2010 ) to guide reporting of highly structured methodologies (e.g., randomized clinical trials); it would be useful to examine researchers’ and editors’ perspectives on the validity of the methodological components in this study for mixed method publications. It is difficult, however, to identify measurable criteria that capture the breadth of study designs in health services. Further, determination of what indicators of rigor would be appropriate might reasonably vary by study design, topic, scope, and even journal, and qualified judgment is required to determine which criteria are appropriate for each study. These findings suggest mixed methods researchers should provide enough detail on methodology and methodological decisions to allow reviewers to judge quality.

Researchers face challenges writing and publishing mixed methods articles, including communicating with diverse audiences who are familiar with only one methodological approach (i.e., quantitative research or qualitative research), determining the most appropriate language and terminology to use, complying with journal word counts, and finding appropriate publishing outlets with reviewers who have expertise in mixed methods research techniques and who are not biased against mixed methods studies ( Leech and Onwuegbuzie 2010 ; Leech, Onwuegbuzie, and Combs 2011 ). Our findings suggest that Sale and Brazil's (2004 ) criteria and existing guidance on conducting mixed methods research (e.g., Collins, Onwuegbuzie, and Sutton 2006 ; Tashakkori and Teddlie 2010 ; Creswell and Plano Clark 2011 ) might be useful frameworks for health services researchers as they work to improve methodological rigor. Journal editors might also encourage the publication of mixed methods projects by (1) publishing guidelines for rigor in mixed methods articles (e.g., Sale and Brazil 2004 ), (2) identifying experienced reviewers who can provide competent and ethical reviews of mixed methods studies, and (3) requiring transparency of methods for all studies so that (4) rigor and quality can be can be assessed to the same extent they are in quantitative studies. These modifications might require (5) some flexibility in word count or allowance of online appendices to allow mixed methods researchers to describe fully and concisely both qualitative and quantitative components, methods for integrating findings, and appropriate details.

Limitations

In this study, assessment was limited to only published articles. We did not contact authors to determine specific study activities, and studies may have included methodological components (e.g., consenting) not reported in publications. We assessed only whether publications reported the methodological component, but we did not evaluate whether each component was fully and appropriately implemented in the research.

Conclusions

Mixed methods studies have utility in providing a more comprehensive picture of health services than either method can alone. Researchers who use mixed methods techniques should use rigorous methodologies in their mixed methods research designs and explicitly report key methodological components of those designs and methods in published articles. Similarly, journal editors who publish mixed methods research should provide guidance to reviewers of mixed methods articles to assess the quality of manuscripts, and they must be prepared to provide adequate space for authors to report the necessary methodological information. Frameworks are now available to guide both the design and evaluation of mixed methods research studies and published works. Whatever frameworks are used, it is essential that authors who engage in mixed methods research studies meet two primary goals (developed by the American Educational Research Association 2006 ): Mixed methods researchers should (1) conduct and report research that is warranted or defensible in terms of documenting evidence, substantiating results, and validating conclusions; and (2) ensure that the conduct of research is transparent in terms of clarifying the logic underpinning the inquiry.

Acknowledgments

Joint Acknowledgment/Disclosure Statement : The authors appreciate funding from the National Institute on Drug Abuse (K23 DA020487) and comments and feedback on an earlier draft from the anonymous reviewers, John Creswell, PhD, Alicia O'Cathain, PhD, Hilary Vidair, PhD, Susan Essock, PhD, and Sa Shen, PhD. Portions of this manuscript were presented at the International Mixed Methods Conference in July 2010 in Baltimore, Maryland.

Disclosures : None.

Disclaimers : None.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

• Aboelela SW, Larson E, Bakken S, Carrasquillo O, Formicola A, Glied SA, Haas J, Gebbie KM. Defining Interdisciplinary Research: Conclusions from a Critical Review of the Literature. Health Services Research. 2007; 42 (1, part 1):329–46. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• American Educational Research Association. Standards for Reporting on Empirical Social Science Research in AERA Publications. Educational Researcher. 2006; 35 (6):33–40. [ Google Scholar ]
• Armstrong R, Waters E, Moore L, Riggs E, Cuervo LG, Lumbiganon P, Hawe P. Improving the Reporting of Public Health Intervention Research: Advancing TREND and CONSORT. Journal of Public Health. 2008; 30 (1):103–9. [ PubMed ] [ Google Scholar ]
• Berwick D. The John Eisenberg Lecture: Health Services Research as a Citizen in Improvement. Health Services Research. 2005; 40 (2):317–36. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bradley EH, Curry LA, Devers KJ. Qualitative Data Analysis for Health Services Research: Developing Taxonomy, Themes, and Theory. Health Services Research. 2007; 42 (4):1758–72. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Brannen J. Mixing Methods: Qualitative and Quantitative Research. Aldershot, England: Ashgate; 1992. [ Google Scholar ]
• Bryman A. Integrating Quantitative and Qualitative Research: How Is It Done? Qualitative Research. 2006; 6 :97–113. [ Google Scholar ]
• Campbell DT. Factors Relevant to the Validity of Experiments in Social Settings. Psychological Bulletin. 1957; 54 :297–312. [ PubMed ] [ Google Scholar ]
• Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Chicago, IL: Rand McNally; 1963. [ Google Scholar ]
• Collins KMT, Onwuegbuzie AJ, Jiao QG. A Mixed Methods Investigation of Mixed Methods Sampling Designs in Social and Health Science Research. Journal of Mixed Methods Research. 2007; 1 :267–94. [ Google Scholar ]
• Collins KMT, Onwuegbuzie AJ, Sutton IL. A Model Incorporating the Rationale and Purpose for Conducting Mixed Methods Research in Special Education and Beyond. Learning Disabilities: A Contemporary Journal. 2006; 4 :67–100. [ Google Scholar ]
• Consolidated Standards of Reporting Trials. 2010. Website [accessed on October 18, 2011]. Available at http://www.consort-statement.org/
• Cousineau N, McDowell I, Hotz S, Hébert P. Measuring Chronic Patients’ Feelings of Being a Burden to Their Caregivers. Medical Care. 2003; 41 :110–8. [ PubMed ] [ Google Scholar ]
• Creswell J. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage; 2008. [ Google Scholar ]
• Creswell JW, Klassen AC, Plano Clark VL, Smith KC for the Office of Behavioral and Social Sciences Research. Best Practices for Mixed Methods Research in the Health Sciences. National Institutes of Health; 2011. [accessed on October 18, 2011]. Available at: http://obssr.od.nih.gov/mixed_methods_research . [ Google Scholar ]
• Creswell J, Plano Clark V. Designing and Conducting Mixed Methods Research. 2nd Edition. Thousand Oaks, CA: Sage; 2011. [ Google Scholar ]
• Creswell JW, Tashakkori A. Developing Publishable Mixed Methods Manuscripts. Journal of Mixed Methods Research. 2008; 1 :107–11. [ Google Scholar ]
• Curry LA, Nembhard IM, Bradley EH. Qualitative and Mixed Methods Provide Unique Contributions to Outcomes Research. Circulation. 2009; 119 :1442–52. [ PubMed ] [ Google Scholar ]
• Dandrove D, Hughes FX, Shanley M. Determinants of HMO Formulary Adoption Decisions. Health Services Research. 2003; 38 (1):169–90. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Denzin NK, Lincoln YS. Handbook of Qualitative Research. Thousand Oaks, CA: Sage; 1994. [ Google Scholar ]
• Devers KJ, Pham HH, Liu G. What Is Driving Hospitals’ Patient-Safety Efforts? Health Affairs. 2004; 23 (2):103–15. [ PubMed ] [ Google Scholar ]
• Felt-Lisk S, Gimm G, Peterson S. Making Pay-For-Performance Work in Medicaid. Health Affairs. 2007; 26 :w516–27. [ PubMed ] [ Google Scholar ]
• Field MJ, Tranquada RE, Feasley JC. Health Services Research: Workforce and Educational Issues. Washington, DC: National Academies Press; 1995. [ Google Scholar ]
• Giacomini MK, Cook DJ for the Evidence-based Medicine Working Group. Users’ Guides to the Medical Literature. XXIII. Qualitative Research in Health Care A. Are the Results of the Study Valid? Journal of the American Medical Association. 2000; 284 :357–62. [ PubMed ] [ Google Scholar ]
• Greene JC, Caracelli VJ, Graham WF. Toward a Conceptual Framework for Mixed-Method Evaluation. Educational Evaluation and Policy. 1989; 11 :255–74. [ Google Scholar ]
• Guba EG, Lincoln YS. Fourth Generation Evaluation. Newbury Park, CA: Sage; 1989. [ Google Scholar ]
• Hall KB, Tett SE, Nissen LM. Perceptions of the Influence of Prescription Medicine Samples on Prescribing by Family Physicians. Medical Care. 2006; 44 (4):383–7. [ PubMed ] [ Google Scholar ]
• Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): Conceptualizing and Measuring Activation in Patients and Consumers. Health Services Research. 2004; 39 (4, part 1):1005–26. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hoadley JF, Cunningham P, McHugh M. Popular Medicaid Programs Do Battle with State Budget Pressures: Perspectives from Twelve States. Health Affairs. 2004; 23 (2):143–54. [ PubMed ] [ Google Scholar ]
• Institute for Scientific Information. 2007. Web of Knowledge Journal Citation Report for Health Care Sciences and Services [accessed on October 18, 2011]. Available at http://admin-apps.isiknowledge.com/JCR/JCR?RQ=LIST_SUMMARY_JOURNAL .
• Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a Definition of Mixed Methods Research. Journal of Mixed Methods Research. 2007; 2 :112–33. [ Google Scholar ]
• Leech NL, Dellinger AB, Brannigan KB, Tanaka H. Evaluating Mixed Research Studies: A Mixed Methods Approach. Journal of Mixed Method Research. 2009; 4 (1):17–31. [ Google Scholar ]
• Leech NL, Onwuegbuzie AJ. A Typology of Mixed Methods Research Designs. Quality & Quantity: International Journal of Methodology. 2009; 43 :265–75. [ Google Scholar ]
• Leech NL, Onwuegbuzie AJ. Guidelines for Conducting and Reporting Mixed Research in the Field of Counseling and Beyond. Journal of Counseling and Development. 2010; 89 (1):61–70. [ Google Scholar ]
• Leech NL, Onwuegbuzie AJ, Combs JP. Writing Publishable Mixed Research Articles: Guidelines for Emerging Scholars in the Health Sciences and Beyond. Mixed Methods Research in the Health Sciences. 2011; 5 (1):7–24. [ Google Scholar ]
• Levine C, Albert SM, Hokenstad A, Halper DE, Hart AY, Gould DA. ‘This Case Is Closed’: Family Caregivers and the Termination of Home Health Care Services for Stroke Patients. Milbank Quarterly. 2006; 84 (2):305–31. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lincoln YS, Guba EG. Naturalistic Inquiry. Beverly Hills, CA: Sage; 1985. [ Google Scholar ]
• Lohr KN, Steinwachs DM. Health Services Research: An Evolving Definition of the Field. Health Services Research. 2002; 37 :15–7. [ PubMed ] [ Google Scholar ]
• Malterud K. Qualitative Research: Standards, Challenges, and Guidelines. Lancet. 2001; 358 :483–8. [ PubMed ] [ Google Scholar ]
• Maxwell JA. Qualitative Research Design: An Interactive Approach. 2nd Edition. Newbury Park, CA: Sage; 2005. [ Google Scholar ]
• McKibbon KA, Gadd CS. A Quantitative Analysis of Qualitative Studies in Clinical Journals for the 2000 Publishing Year. BioMed Central Medical Informatics and Decision Making. 2004; 4 (11) [accessed on October 18, 2011]. Available at http://www.biomedcentral.com/1472-6947/4/11 . [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Messick S. Validity. In: Linn RL, editor. Educational Measurement. 3rd Edition. Old Tappan, NJ: Macmillan; 1989. pp. 13–103. [ Google Scholar ]
• Messick S. Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. American Psychologist. 1995; 50 :741–9. [ Google Scholar ]
• Miles M, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. 2nd Edition. Thousand Oaks, CA: Sage; 1994. [ Google Scholar ]
• O'Cathain A. Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. In: Tashakkori A, Teddlie C, editors. Sage Handbook of Mixed Methods in Social and Behavioral Research. 2nd Edition. Thousand Oaks, CA: Sage; 2010. pp. 531–57. [ Google Scholar ]
• O'Cathain A, Murphy E, Nicholl J. Integration and Publications as Indicators of ‘Yield’ from Mixed Methods Studies. Journal of Mixed Methods Research. 2007; 1 (2):147–63. [ Google Scholar ]
• Onwuegbuzie AJ. Expanding the Framework of Internal and External Validity in Quantitative Research. Research in the Schools. 2003; 10 (1):71–90. [ Google Scholar ]
• Onwuegbuzie AJ, Collins KMT. A Typology of Mixed Methods Sampling Designs in Social Science Research. Qualitative Report. 2007; 12 :281–316. [ Google Scholar ]
• Onwuegbuzie AJ, Daniel LG. A Framework for Reporting and Interpreting Internal Consistency Reliability Estimates. Measurement and Evaluation in Counseling and Development. 2002; 35 :89–103. [ Google Scholar ]
• Onwuegbuzie AJ, Daniel LG. Reliability Generalization: The Importance of Considering Sample Specificity, Confidence Intervals, and Subgroup Differences. Research in the Schools. 2004; 11 (1):61–72. [ Google Scholar ]
• Padgett D. Qualitative Methods in Social Work Research. 2nd Edition. Thousand Oaks, CA: Sage; 2008. [ Google Scholar ]
• Pbert L, Vukovic N, Ockene JK, Hollis JF, Riedlinger K. Developing and Testing New Smoking Measures for the Health Plan Employer Data and Information Set. Medical Care. 2003; 41 (4):550–9. [ PubMed ] [ Google Scholar ]
• Plano Clark VL. The Adoption and Practice of Mixed Methods: U.S. Trends in Federally Funded Health-Related Research. Qualitative Inquiry. 2010; 16 :428–40. [ Google Scholar ]
• Pluye P, Gagnon M, Griffiths F, Johnson-Lafleur J. A Scoring System for Appraising Mixed Methods Research and Concomitantly Appraising Qualitative, Quantitative and Mixed Methods Primary Studies in Mixed Studies Reviews. International Journal of Nursing Studies. 2009; 46 :529–46. [ PubMed ] [ Google Scholar ]
• Pope C, Mays N. Qualitative Research in Health Care. 3rd Edition. Malden, MA: Blackwell Publishing; 2006. [ Google Scholar ]
• Powell H, Mihalas S, Onwuegbuzie AJ, Suldo S, Daley CE. Mixed Methods Research in School Psychology: A Mixed Methods Investigation of Trends in the Literature. Psychology in the Schools. 2008; 45 :291–309. [ Google Scholar ]
• Rosenthal MB, Landon BE, Howitt K, Song HR, Epstein AM. Climbing Up the Pay-For-Performance Learning Curve: Where Are The Early Adopters Now? Health Affairs. 2007; 26 (6):1674–82. [ PubMed ] [ Google Scholar ]
• Sale JEM, Brazil K. A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies. Quality and Quantity. 2004; 38 (4):351–65. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Schore J, Foster L, Phillips B. Consumer Enrollment and Experiences in the Cash and Counseling Program. Health Services Research. 2007; 42 (1, part 2):446–66. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shortell S. The Emergence of Qualitative Methods in Health Services Research. Health Services Research. 1999; 34 :1083–90. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Tashakkori A, Teddlie C. Sage Handbook of Mixed Methods in Social and Behavioral Research. 2nd Edition. Thousand Oaks, CA: Sage; 2010. [ Google Scholar ]
• Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: Sage; 2009. [ Google Scholar ]