what makes a research study good

What Makes a Good Research Study?

Find out what separates a solid research study from a so-so one..

Posted March 31, 2018

One day you read online that drinking coffee reduces the chances of having age-related memory decline . You start drinking coffee. The next month you read that drinking coffee increases your chances of having age-related memory decline. What gives? In this article, you will learn how to "decipher" research studies to figure out what a research study is really saying - and what it doesn't say. You'll also discover how to tell if the reporting on a particular study was accurate or not. Ask yourself the following six questions when looking at a research study. Keep in mind these are just six of the many factors that make up a "clean" study.

1. Did the study use a placebo , and were the staff blinded to treatment?

The brain is very susceptible to placebos. There is evidence that even when you tell study subjects (participants) that they are getting a placebo, they improve (Carvalho, et al., 2016). In pharmaceutical studies, the US Food and Drug Administration (FDA) requires pharmaceutical companies to do double-blind placebo-controlled studies. This means that the study subjects, the physicians dispensing the drug, and the clinicians rating the subjects' behavior don't know what subjects are getting - drug or placebo. This eliminates a lot of bias , and it helps show whether the drug actually works.

2. Was there a bogus/sham treatment?

A bogus/sham treatment is one in which subjects are given a treatment that looks very much like the real treatment, except for one major difference. The bogus/sham treatment doesn't actually provide the therapeutic part of the treatment. For example, some acupuncture studies use a sham/bogus treatment, such as a 2017 study by Ugurlu, et al. regarding acupuncture treatment for fibromyalgia .

Bogus/sham treatments, when compared to active treatments, help researchers discover whether the active treatment is what works, or the fact that people think they are getting the active treatment.

3. How many people were there in the study (N)?

Logic says the more people you have in a study, or the study's "N", the better chance you have of your study representing of the general population (the "generalizability" of a study). Let's say you're studying the effects of apple juice on ADHD symptoms, and you have a total N of ten people. By chance, seven of those ten people have severe ADHD, two have moderate ADHD, and one has mild ADHD. You now could throw off the results of your study because you have so many people with severe ADHD in the study. When you have more subjects, or a larger N, in a study, there is more of a chance that you would have people that have mild, moderate, and severe ADHD.

4. Were the study groups randomized?

A good study randomizes their subjects into the active treatment and placebo groups. This means that the subjects are in those particular groups by chance. This provides extra "backup" that the effects from a treatment were actually from that treatment, not from study staff bias.

5. Who conducted the research, and who is paying for it?

If the people that created a treatment are also testing a treatment, this is a concern. When you have a horse in the race, so to speak, it is more difficult to be unbiased. Another concern is if an entity with a vested interest in a particular study outcome is paying for that study. For example, if there is a study on the effectiveness of widgets, and the sole source of funding is Widgets are Wonderful, Inc., and the researchers are employees of Widgets are Wonderful, that study better have some seriously good methodology to help eliminate bias. Even better, an independent research group is funded by an organization without ties to the study outcome.

6. Was the article published in a refereed (peer-reviewed/scholarly) journal?

In a refereed journal, a manuscript is reviewed by other experts in the field before it is published as an article. The authors of the manuscript are not disclosed to the reviewers, in order to reduce possible bias. When we review manuscripts for a journal, there are three main categories: reject, meaning the article goes no further; accept, with revisions, meaning the authors must edit their article before resubmitting it for publication; and accept as written, which is rare, but once in a while there is a manuscript with such good research methodology and writing that no additional editing is needed.

When a journal is not refereed, the standard of inclusion into that journal is not as high. This means the quality of the research may not be up to the same standards. Look up the journal online to find out if it is a peer-reviewed journal.

If you don't have university access, you can at least access the abstracts of journal articles at Google Scholar . The abstract lets you know the study's methodology, the number of study subjects, the outcomes, and the author's conclusions.

You may also see the term "open-access" used to describe a journal. An open-access journal is one that users can freely access, without a subscription or fees. Some open-access journals are peer-reviewed, some are not.

Copyright 2018 Sarkis Media

Carvalho, C., Caetano, J. M., Cunha, L., Rebouta, P., Kaptchuk, T. J., & Kirsch, I. (2016). Open-label Placebo Treatment in Chronic Low Back Pain: A Randomized Controlled Trial. Pain, 157(12), 2766–2772. http://doi.org/10.1097/j.pain.0000000000000700

Uğurlu, F. G., Sezer, N., Aktekin, L., Fidan, F., Tok, F., & Akkuş, S. (2017). The effects of acupuncture versus sham acupuncture in the treatment of fibromyalgia: a randomized controlled clinical trial. Acta reumatologica portuguesa, (1).

Stephanie Moulton Sarkis, Ph.D., N.C.C., D.C.M.H.S., L.M.H.C ., is the author of Gaslighting: Recognize Manipulative and Emotionally Abusive People — and Break Free .

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

At any moment, someone’s aggravating behavior or our own bad luck can set us off on an emotional spiral that threatens to derail our entire day. Here’s how we can face our triggers with less reactivity so that we can get on with our lives.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience

15 Steps to Good Research

• Define and articulate a research question (formulate a research hypothesis). How to Write a Thesis Statement (Indiana University)
• Identify possible sources of information in many types and formats. Georgetown University Library's Research & Course Guides
• Judge the scope of the project.
• Reevaluate the research question based on the nature and extent of information available and the parameters of the research project.
• Select the most appropriate investigative methods (surveys, interviews, experiments) and research tools (periodical indexes, databases, websites).
• Plan the research project. Writing Anxiety (UNC-Chapel Hill) Strategies for Academic Writing (SUNY Empire State College)
• Retrieve information using a variety of methods (draw on a repertoire of skills).
• Refine the search strategy as necessary.
• Write and organize useful notes and keep track of sources. Taking Notes from Research Reading (University of Toronto) Use a citation manager: Zotero or Refworks
• Evaluate sources using appropriate criteria. Evaluating Internet Sources
• Synthesize, analyze and integrate information sources and prior knowledge. Georgetown University Writing Center
• Revise hypothesis as necessary.
• Use information effectively for a specific purpose.
• Understand such issues as plagiarism, ownership of information (implications of copyright to some extent), and costs of information. Georgetown University Honor Council Copyright Basics (Purdue University) How to Recognize Plagiarism: Tutorials and Tests from Indiana University
• Cite properly and give credit for sources of ideas. MLA Bibliographic Form (7th edition, 2009) MLA Bibliographic Form (8th edition, 2016) Turabian Bibliographic Form: Footnote/Endnote Turabian Bibliographic Form: Parenthetical Reference Use a citation manager: Zotero or Refworks

Adapted from the Association of Colleges and Research Libraries "Objectives for Information Literacy Instruction" , which are more complete and include outcomes. See also the broader "Information Literacy Competency Standards for Higher Education."

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

82k Accesses

28 Citations

72 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

Good Qualitative Research: Opening up the Debate

Beyond qualitative/quantitative structuralism: the positivist qualitative research and the paradigmatic disclaimer.

What is Qualitative in Research

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

What Constitutes a Good Research?

The Declining Art of Good Research

We seem to be compromising our commitment to good research in favor of publishable research, and there are a combination of trends that are accountable for this.

The first is the continued pressure of “publish or perish” for young academics seeking to move forward on the track for fewer and fewer tenured positions (or increasingly draconian renewable contracts).

Secondly, the open access model of research publication has created a booming population of academic journals with pages to fill and new researchers willing to pay article publication fees (APFs).

Thirdly, budget-strapped institutions have been aggressively targeting doctoral research candidates and the higher fees they bring to the table.

When these three trends are combined, the resulting onslaught of quantity over quality leads us to question what “good” research looks like anymore.

Is it the institution from which the research originated, or the debatable rank of the journal that published it?

Good Research as a Methodological Question

When looking to learn how to recognize what “good” research looks like, it makes sense to start at the beginning with the basic scope of the project:

• Does the research have a solid hypothesis?
• Is there evidence of a comprehensive literature review from reputable sources that clearly defines a target area for valuable research?
• Is the research team allocating sufficient time/resources to do the job properly, or were compromises made in order to accommodate the available funding?
• Is there evidence of a willingness to refine the hypothesis and research strategy if needed?
• Are the expectations of the implications of the research realistic?

Characteristics of a Good Research

For conducting a systematic research, it is important understand the characteristics of a good research.

• Its relevance to existing research conducted by other researchers.
• A good research is doable and replicable in future.
• It must be based on a logical rationale and tied to theory.
• It must generate new questions or hypotheses for incremental work in future.
• It must directly or indirectly address some real world problem.
• It must clearly state the variables of the experiment.
• It must conclude with valid and verifiable findings.

Good Research as an Ethical Question

The question as to whether or not the research is worth conducting at all could generate an extended and heated debate. Researchers are expected to publish, and research budgets are there to be spent.

We can hope that there was some degree of discussion and oversight before the research project was given the green light by a Principal Investigator or Research Supervisor, but those decisions are often made in a context of simple obligation rather than perceived need.

Consider the example of a less than proactive doctoral student with limited time and resources to complete a dissertation topic. A suggestion is made by the departmental Research Supervisor to pick a dissertation from a decade ago and simply repeat it. The suggestion meets the need for expediency and simplicity, but raises as many questions as it answers:

• What is the validity of the study – just because it can be repeated, should it?
• What was the contribution of the original study to the general body of knowledge? Will this additional data be an improvement?
• Given the lack of interest among academic journals in replicated studies, is the suggestion denying the student the opportunity to get published?
• Is directing a student to replication in the interests of expediency meeting a broader academic goal of graduating proficient researchers?

The Building Blocks of “Good” Research

There is no shortage of reputable, peer-reviewed journals that publish first-rate research material for new researchers to model.

That doesn’t mean you should copy the research topic or the methodology, but it wouldn’t hurt to examine the protocol in detail and make note of the specific decisions made and criteria put in place when that protocol was developed and implemented.

The challenge lies in sticking to those tried-and-true methodologies when your research data doesn’t prove to be as rich and fruitful as you had hoped.

Have you ever been stuck while in the middle of conducting a research? How did you cope with that? Let us know your approach while conducting a good research in the comments section below!

You can also visit our  Q&A forum  for frequently asked questions related to different aspects of research writing and publishing answered by our team that comprises subject-matter experts, eminent researchers, and publication experts.

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

Ensuring Academic Integrity and Transparency in Academic Research: A comprehensive checklist for researchers

Academic integrity is the foundation upon which the credibility and value of scientific findings are…

• AI in Academia

AI vs. AI: How to detect image manipulation and avoid academic misconduct

The scientific community is facing a new frontier of controversy as artificial intelligence (AI) is…

• Diversity and Inclusion

Need for Diversifying Academic Curricula: Embracing missing voices and marginalized perspectives

In classrooms worldwide, a single narrative often dominates, leaving many students feeling lost. These stories,…

• Career Corner
• Trending Now

Recognizing the signs: A guide to overcoming academic burnout

As the sun set over the campus, casting long shadows through the library windows, Alex…

Reassessing the Lab Environment to Create an Equitable and Inclusive Space

The pursuit of scientific discovery has long been fueled by diverse minds and perspectives. Yet…

7 Steps of Writing an Excellent Academic Book Chapter

When Your Thesis Advisor Asks You to Quit

Virtual Defense: Top 5 Online Thesis Defense Tips

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

As a researcher, what do you consider most when choosing an image manipulation detector?

What makes a high quality clinical research paper?

Affiliation.

• 1 BMJ, London, UK. [email protected]
• PMID: 20233312
• DOI: 10.1111/j.1601-0825.2010.01663.x

The quality of a research paper depends primarily on the quality of the research study it reports. However, there is also much that authors can do to maximise the clarity and usefulness of their papers. Journals' instructions for authors often focus on the format, style, and length of articles but do not always emphasise the need to clearly explain the work's science and ethics: so this review reminds researchers that transparency is important too. The research question should be stated clearly, along with an explanation of where it came from and why it is important. The study methods must be reported fully and, where appropriate, in line with an evidence based reporting guideline such as the CONSORT statement for randomised controlled trials. If the study was a trial the paper should state where and when the study was registered and state its registration identifier. Finally, any relevant conflicts of interest should be declared.

Publication types

• Clinical Trials as Topic*
• Ethics, Research*
• Guidelines as Topic
• Journalism, Medical / standards*
• Periodicals as Topic
• Publishing / standards*
• Writing / standards*

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Turk J Anaesthesiol Reanim
• v.44(4); 2016 Aug

What is Scientific Research and How Can it be Done?

Scientific researches are studies that should be systematically planned before performing them. In this review, classification and description of scientific studies, planning stage randomisation and bias are explained.

Research conducted for the purpose of contributing towards science by the systematic collection, interpretation and evaluation of data and that, too, in a planned manner is called scientific research: a researcher is the one who conducts this research. The results obtained from a small group through scientific studies are socialised, and new information is revealed with respect to diagnosis, treatment and reliability of applications. The purpose of this review is to provide information about the definition, classification and methodology of scientific research.

Before beginning the scientific research, the researcher should determine the subject, do planning and specify the methodology. In the Declaration of Helsinki, it is stated that ‘the primary purpose of medical researches on volunteers is to understand the reasons, development and effects of diseases and develop protective, diagnostic and therapeutic interventions (method, operation and therapies). Even the best proven interventions should be evaluated continuously by investigations with regard to reliability, effectiveness, efficiency, accessibility and quality’ ( 1 ).

The questions, methods of response to questions and difficulties in scientific research may vary, but the design and structure are generally the same ( 2 ).

Classification of Scientific Research

Scientific research can be classified in several ways. Classification can be made according to the data collection techniques based on causality, relationship with time and the medium through which they are applied.

• Observational
• Experimental
• Descriptive
• Retrospective
• Prospective
• Cross-sectional
• Social descriptive research ( 3 )

Another method is to classify the research according to its descriptive or analytical features. This review is written according to this classification method.

I. Descriptive research

• Case series
• Surveillance studies

II. Analytical research

• Observational studies: cohort, case control and cross- sectional research
• Interventional research: quasi-experimental and clinical research
• Case Report: it is the most common type of descriptive study. It is the examination of a single case having a different quality in the society, e.g. conducting general anaesthesia in a pregnant patient with mucopolysaccharidosis.
• Case Series: it is the description of repetitive cases having common features. For instance; case series involving interscapular pain related to neuraxial labour analgesia. Interestingly, malignant hyperthermia cases are not accepted as case series since they are rarely seen during historical development.
• Surveillance Studies: these are the results obtained from the databases that follow and record a health problem for a certain time, e.g. the surveillance of cross-infections during anaesthesia in the intensive care unit.

Moreover, some studies may be experimental. After the researcher intervenes, the researcher waits for the result, observes and obtains data. Experimental studies are, more often, in the form of clinical trials or laboratory animal trials ( 2 ).

Analytical observational research can be classified as cohort, case-control and cross-sectional studies.

Firstly, the participants are controlled with regard to the disease under investigation. Patients are excluded from the study. Healthy participants are evaluated with regard to the exposure to the effect. Then, the group (cohort) is followed-up for a sufficient period of time with respect to the occurrence of disease, and the progress of disease is studied. The risk of the healthy participants getting sick is considered an incident. In cohort studies, the risk of disease between the groups exposed and not exposed to the effect is calculated and rated. This rate is called relative risk. Relative risk indicates the strength of exposure to the effect on the disease.

Cohort research may be observational and experimental. The follow-up of patients prospectively is called a prospective cohort study . The results are obtained after the research starts. The researcher’s following-up of cohort subjects from a certain point towards the past is called a retrospective cohort study . Prospective cohort studies are more valuable than retrospective cohort studies: this is because in the former, the researcher observes and records the data. The researcher plans the study before the research and determines what data will be used. On the other hand, in retrospective studies, the research is made on recorded data: no new data can be added.

In fact, retrospective and prospective studies are not observational. They determine the relationship between the date on which the researcher has begun the study and the disease development period. The most critical disadvantage of this type of research is that if the follow-up period is long, participants may leave the study at their own behest or due to physical conditions. Cohort studies that begin after exposure and before disease development are called ambidirectional studies . Public healthcare studies generally fall within this group, e.g. lung cancer development in smokers.

• Case-Control Studies: these studies are retrospective cohort studies. They examine the cause and effect relationship from the effect to the cause. The detection or determination of data depends on the information recorded in the past. The researcher has no control over the data ( 2 ).

Cross-sectional studies are advantageous since they can be concluded relatively quickly. It may be difficult to obtain a reliable result from such studies for rare diseases ( 2 ).

Cross-sectional studies are characterised by timing. In such studies, the exposure and result are simultaneously evaluated. While cross-sectional studies are restrictedly used in studies involving anaesthesia (since the process of exposure is limited), they can be used in studies conducted in intensive care units.

• Quasi-Experimental Research: they are conducted in cases in which a quick result is requested and the participants or research areas cannot be randomised, e.g. giving hand-wash training and comparing the frequency of nosocomial infections before and after hand wash.
• Clinical Research: they are prospective studies carried out with a control group for the purpose of comparing the effect and value of an intervention in a clinical case. Clinical study and research have the same meaning. Drugs, invasive interventions, medical devices and operations, diets, physical therapy and diagnostic tools are relevant in this context ( 6 ).

Clinical studies are conducted by a responsible researcher, generally a physician. In the research team, there may be other healthcare staff besides physicians. Clinical studies may be financed by healthcare institutes, drug companies, academic medical centres, volunteer groups, physicians, healthcare service providers and other individuals. They may be conducted in several places including hospitals, universities, physicians’ offices and community clinics based on the researcher’s requirements. The participants are made aware of the duration of the study before their inclusion. Clinical studies should include the evaluation of recommendations (drug, device and surgical) for the treatment of a disease, syndrome or a comparison of one or more applications; finding different ways for recognition of a disease or case and prevention of their recurrence ( 7 ).

Clinical Research

In this review, clinical research is explained in more detail since it is the most valuable study in scientific research.

Clinical research starts with forming a hypothesis. A hypothesis can be defined as a claim put forward about the value of a population parameter based on sampling. There are two types of hypotheses in statistics.

• H 0 hypothesis is called a control or null hypothesis. It is the hypothesis put forward in research, which implies that there is no difference between the groups under consideration. If this hypothesis is rejected at the end of the study, it indicates that a difference exists between the two treatments under consideration.
• H 1 hypothesis is called an alternative hypothesis. It is hypothesised against a null hypothesis, which implies that a difference exists between the groups under consideration. For example, consider the following hypothesis: drug A has an analgesic effect. Control or null hypothesis (H 0 ): there is no difference between drug A and placebo with regard to the analgesic effect. The alternative hypothesis (H 1 ) is applicable if a difference exists between drug A and placebo with regard to the analgesic effect.

The planning phase comes after the determination of a hypothesis. A clinical research plan is called a protocol . In a protocol, the reasons for research, number and qualities of participants, tests to be applied, study duration and what information to be gathered from the participants should be found and conformity criteria should be developed.

The selection of participant groups to be included in the study is important. Inclusion and exclusion criteria of the study for the participants should be determined. Inclusion criteria should be defined in the form of demographic characteristics (age, gender, etc.) of the participant group and the exclusion criteria as the diseases that may influence the study, age ranges, cases involving pregnancy and lactation, continuously used drugs and participants’ cooperation.

The next stage is methodology. Methodology can be grouped under subheadings, namely, the calculation of number of subjects, blinding (masking), randomisation, selection of operation to be applied, use of placebo and criteria for stopping and changing the treatment.

I. Calculation of the Number of Subjects

The entire source from which the data are obtained is called a universe or population . A small group selected from a certain universe based on certain rules and which is accepted to highly represent the universe from which it is selected is called a sample and the characteristics of the population from which the data are collected are called variables. If data is collected from the entire population, such an instance is called a parameter . Conducting a study on the sample rather than the entire population is easier and less costly. Many factors influence the determination of the sample size. Firstly, the type of variable should be determined. Variables are classified as categorical (qualitative, non-numerical) or numerical (quantitative). Individuals in categorical variables are classified according to their characteristics. Categorical variables are indicated as nominal and ordinal (ordered). In nominal variables, the application of a category depends on the researcher’s preference. For instance, a female participant can be considered first and then the male participant, or vice versa. An ordinal (ordered) variable is ordered from small to large or vice versa (e.g. ordering obese patients based on their weights-from the lightest to the heaviest or vice versa). A categorical variable may have more than one characteristic: such variables are called binary or dichotomous (e.g. a participant may be both female and obese).

If the variable has numerical (quantitative) characteristics and these characteristics cannot be categorised, then it is called a numerical variable. Numerical variables are either discrete or continuous. For example, the number of operations with spinal anaesthesia represents a discrete variable. The haemoglobin value or height represents a continuous variable.

Statistical analyses that need to be employed depend on the type of variable. The determination of variables is necessary for selecting the statistical method as well as software in SPSS. While categorical variables are presented as numbers and percentages, numerical variables are represented using measures such as mean and standard deviation. It may be necessary to use mean in categorising some cases such as the following: even though the variable is categorical (qualitative, non-numerical) when Visual Analogue Scale (VAS) is used (since a numerical value is obtained), it is classified as a numerical variable: such variables are averaged.

Clinical research is carried out on the sample and generalised to the population. Accordingly, the number of samples should be correctly determined. Different sample size formulas are used on the basis of the statistical method to be used. When the sample size increases, error probability decreases. The sample size is calculated based on the primary hypothesis. The determination of a sample size before beginning the research specifies the power of the study. Power analysis enables the acquisition of realistic results in the research, and it is used for comparing two or more clinical research methods.

Because of the difference in the formulas used in calculating power analysis and number of samples for clinical research, it facilitates the use of computer programs for making calculations.

It is necessary to know certain parameters in order to calculate the number of samples by power analysis.

• Type-I (α) and type-II (β) error levels
• Difference between groups (d-difference) and effect size (ES)
• Distribution ratio of groups
• Direction of research hypothesis (H1)

a. Type-I (α) and Type-II (β) Error (β) Levels

Two types of errors can be made while accepting or rejecting H 0 hypothesis in a hypothesis test. Type-I error (α) level is the probability of finding a difference at the end of the research when there is no difference between the two applications. In other words, it is the rejection of the hypothesis when H 0 is actually correct and it is known as α error or p value. For instance, when the size is determined, type-I error level is accepted as 0.05 or 0.01.

Another error that can be made during a hypothesis test is a type-II error. It is the acceptance of a wrongly hypothesised H 0 hypothesis. In fact, it is the probability of failing to find a difference when there is a difference between the two applications. The power of a test is the ability of that test to find a difference that actually exists. Therefore, it is related to the type-II error level.

Since the type-II error risk is expressed as β, the power of the test is defined as 1–β. When a type-II error is 0.20, the power of the test is 0.80. Type-I (α) and type-II (β) errors can be intentional. The reason to intentionally make such an error is the necessity to look at the events from the opposite perspective.

b. Difference between Groups and ES

ES is defined as the state in which statistical difference also has clinically significance: ES≥0.5 is desirable. The difference between groups is the absolute difference between the groups compared in clinical research.

c. Allocation Ratio of Groups

The allocation ratio of groups is effective in determining the number of samples. If the number of samples is desired to be determined at the lowest level, the rate should be kept as 1/1.

d. Direction of Hypothesis (H1)

The direction of hypothesis in clinical research may be one-sided or two-sided. While one-sided hypotheses hypothesis test differences in the direction of size, two-sided hypotheses hypothesis test differences without direction. The power of the test in two-sided hypotheses is lower than one-sided hypotheses.

After these four variables are determined, they are entered in the appropriate computer program and the number of samples is calculated. Statistical packaged software programs such as Statistica, NCSS and G-Power may be used for power analysis and calculating the number of samples. When the samples size is calculated, if there is a decrease in α, difference between groups, ES and number of samples, then the standard deviation increases and power decreases. The power in two-sided hypothesis is lower. It is ethically appropriate to consider the determination of sample size, particularly in animal experiments, at the beginning of the study. The phase of the study is also important in the determination of number of subjects to be included in drug studies. Usually, phase-I studies are used to determine the safety profile of a drug or product, and they are generally conducted on a few healthy volunteers. If no unacceptable toxicity is detected during phase-I studies, phase-II studies may be carried out. Phase-II studies are proof-of-concept studies conducted on a larger number (100–500) of volunteer patients. When the effectiveness of the drug or product is evident in phase-II studies, phase-III studies can be initiated. These are randomised, double-blinded, placebo or standard treatment-controlled studies. Volunteer patients are periodically followed-up with respect to the effectiveness and side effects of the drug. It can generally last 1–4 years and is valuable during licensing and releasing the drug to the general market. Then, phase-IV studies begin in which long-term safety is investigated (indication, dose, mode of application, safety, effectiveness, etc.) on thousands of volunteer patients.

II. Blinding (Masking) and Randomisation Methods

When the methodology of clinical research is prepared, precautions should be taken to prevent taking sides. For this reason, techniques such as randomisation and blinding (masking) are used. Comparative studies are the most ideal ones in clinical research.

Blinding Method

A case in which the treatments applied to participants of clinical research should be kept unknown is called the blinding method . If the participant does not know what it receives, it is called a single-blind study; if even the researcher does not know, it is called a double-blind study. When there is a probability of knowing which drug is given in the order of application, when uninformed staff administers the drug, it is called in-house blinding. In case the study drug is known in its pharmaceutical form, a double-dummy blinding test is conducted. Intravenous drug is given to one group and a placebo tablet is given to the comparison group; then, the placebo tablet is given to the group that received the intravenous drug and intravenous drug in addition to placebo tablet is given to the comparison group. In this manner, each group receives both the intravenous and tablet forms of the drug. In case a third party interested in the study is involved and it also does not know about the drug (along with the statistician), it is called third-party blinding.

Randomisation Method

The selection of patients for the study groups should be random. Randomisation methods are used for such selection, which prevent conscious or unconscious manipulations in the selection of patients ( 8 ).

No factor pertaining to the patient should provide preference of one treatment to the other during randomisation. This characteristic is the most important difference separating randomised clinical studies from prospective and synchronous studies with experimental groups. Randomisation strengthens the study design and enables the determination of reliable scientific knowledge ( 2 ).

The easiest method is simple randomisation, e.g. determination of the type of anaesthesia to be administered to a patient by tossing a coin. In this method, when the number of samples is kept high, a balanced distribution is created. When the number of samples is low, there will be an imbalance between the groups. In this case, stratification and blocking have to be added to randomisation. Stratification is the classification of patients one or more times according to prognostic features determined by the researcher and blocking is the selection of a certain number of patients for each stratification process. The number of stratification processes should be determined at the beginning of the study.

As the number of stratification processes increases, performing the study and balancing the groups become difficult. For this reason, stratification characteristics and limitations should be effectively determined at the beginning of the study. It is not mandatory for the stratifications to have equal intervals. Despite all the precautions, an imbalance might occur between the groups before beginning the research. In such circumstances, post-stratification or restandardisation may be conducted according to the prognostic factors.

The main characteristic of applying blinding (masking) and randomisation is the prevention of bias. Therefore, it is worthwhile to comprehensively examine bias at this stage.

Bias and Chicanery

While conducting clinical research, errors can be introduced voluntarily or involuntarily at a number of stages, such as design, population selection, calculating the number of samples, non-compliance with study protocol, data entry and selection of statistical method. Bias is taking sides of individuals in line with their own decisions, views and ideological preferences ( 9 ). In order for an error to lead to bias, it has to be a systematic error. Systematic errors in controlled studies generally cause the results of one group to move in a different direction as compared to the other. It has to be understood that scientific research is generally prone to errors. However, random errors (or, in other words, ‘the luck factor’-in which bias is unintended-do not lead to bias ( 10 ).

Another issue, which is different from bias, is chicanery. It is defined as voluntarily changing the interventions, results and data of patients in an unethical manner or copying data from other studies. Comparatively, bias may not be done consciously.

In case unexpected results or outliers are found while the study is analysed, if possible, such data should be re-included into the study since the complete exclusion of data from a study endangers its reliability. In such a case, evaluation needs to be made with and without outliers. It is insignificant if no difference is found. However, if there is a difference, the results with outliers are re-evaluated. If there is no error, then the outlier is included in the study (as the outlier may be a result). It should be noted that re-evaluation of data in anaesthesiology is not possible.

Statistical evaluation methods should be determined at the design stage so as not to encounter unexpected results in clinical research. The data should be evaluated before the end of the study and without entering into details in research that are time-consuming and involve several samples. This is called an interim analysis . The date of interim analysis should be determined at the beginning of the study. The purpose of making interim analysis is to prevent unnecessary cost and effort since it may be necessary to conclude the research after the interim analysis, e.g. studies in which there is no possibility to validate the hypothesis at the end or the occurrence of different side effects of the drug to be used. The accuracy of the hypothesis and number of samples are compared. Statistical significance levels in interim analysis are very important. If the data level is significant, the hypothesis is validated even if the result turns out to be insignificant after the date of the analysis.

Another important point to be considered is the necessity to conclude the participants’ treatment within the period specified in the study protocol. When the result of the study is achieved earlier and unexpected situations develop, the treatment is concluded earlier. Moreover, the participant may quit the study at its own behest, may die or unpredictable situations (e.g. pregnancy) may develop. The participant can also quit the study whenever it wants, even if the study has not ended ( 7 ).

In case the results of a study are contrary to already known or expected results, the expected quality level of the study suggesting the contradiction may be higher than the studies supporting what is known in that subject. This type of bias is called confirmation bias. The presence of well-known mechanisms and logical inference from them may create problems in the evaluation of data. This is called plausibility bias.

Another type of bias is expectation bias. If a result different from the known results has been achieved and it is against the editor’s will, it can be challenged. Bias may be introduced during the publication of studies, such as publishing only positive results, selection of study results in a way to support a view or prevention of their publication. Some editors may only publish research that extols only the positive results or results that they desire.

Bias may be introduced for advertisement or economic reasons. Economic pressure may be applied on the editor, particularly in the cases of studies involving drugs and new medical devices. This is called commercial bias.

In recent years, before beginning a study, it has been recommended to record it on the Web site www.clinicaltrials.gov for the purpose of facilitating systematic interpretation and analysis in scientific research, informing other researchers, preventing bias, provision of writing in a standard format, enhancing contribution of research results to the general literature and enabling early intervention of an institution for support. This Web site is a service of the US National Institutes of Health.

The last stage in the methodology of clinical studies is the selection of intervention to be conducted. Placebo use assumes an important place in interventions. In Latin, placebo means ‘I will be fine’. In medical literature, it refers to substances that are not curative, do not have active ingredients and have various pharmaceutical forms. Although placebos do not have active drug characteristic, they have shown effective analgesic characteristics, particularly in algology applications; further, its use prevents bias in comparative studies. If a placebo has a positive impact on a participant, it is called the placebo effect ; on the contrary, if it has a negative impact, it is called the nocebo effect . Another type of therapy that can be used in clinical research is sham application. Although a researcher does not cure the patient, the researcher may compare those who receive therapy and undergo sham. It has been seen that sham therapies also exhibit a placebo effect. In particular, sham therapies are used in acupuncture applications ( 11 ). While placebo is a substance, sham is a type of clinical application.

Ethically, the patient has to receive appropriate therapy. For this reason, if its use prevents effective treatment, it causes great problem with regard to patient health and legalities.

Before medical research is conducted with human subjects, predictable risks, drawbacks and benefits must be evaluated for individuals or groups participating in the study. Precautions must be taken for reducing the risk to a minimum level. The risks during the study should be followed, evaluated and recorded by the researcher ( 1 ).

After the methodology for a clinical study is determined, dealing with the ‘Ethics Committee’ forms the next stage. The purpose of the ethics committee is to protect the rights, safety and well-being of volunteers taking part in the clinical research, considering the scientific method and concerns of society. The ethics committee examines the studies presented in time, comprehensively and independently, with regard to ethics and science; in line with the Declaration of Helsinki and following national and international standards concerning ‘Good Clinical Practice’. The method to be followed in the formation of the ethics committee should be developed without any kind of prejudice and to examine the applications with regard to ethics and science within the framework of the ethics committee, Regulation on Clinical Trials and Good Clinical Practice ( www.iku.com ). The necessary documents to be presented to the ethics committee are research protocol, volunteer consent form, budget contract, Declaration of Helsinki, curriculum vitae of researchers, similar or explanatory literature samples, supporting institution approval certificate and patient follow-up form.

Only one sister/brother, mother, father, son/daughter and wife/husband can take charge in the same ethics committee. A rector, vice rector, dean, deputy dean, provincial healthcare director and chief physician cannot be members of the ethics committee.

Members of the ethics committee can work as researchers or coordinators in clinical research. However, during research meetings in which members of the ethics committee are researchers or coordinators, they must leave the session and they cannot sign-off on decisions. If the number of members in the ethics committee for a particular research is so high that it is impossible to take a decision, the clinical research is presented to another ethics committee in the same province. If there is no ethics committee in the same province, an ethics committee in the closest settlement is found.

Thereafter, researchers need to inform the participants using an informed consent form. This form should explain the content of clinical study, potential benefits of the study, alternatives and risks (if any). It should be easy, comprehensible, conforming to spelling rules and written in plain language understandable by the participant.

This form assists the participants in taking a decision regarding participation in the study. It should aim to protect the participants. The participant should be included in the study only after it signs the informed consent form; the participant can quit the study whenever required, even when the study has not ended ( 7 ).

Peer-review: Externally peer-reviewed.

Author Contributions: Concept - C.Ö.Ç., A.D.; Design - C.Ö.Ç.; Supervision - A.D.; Resource - C.Ö.Ç., A.D.; Materials - C.Ö.Ç., A.D.; Analysis and/or Interpretation - C.Ö.Ç., A.D.; Literature Search - C.Ö.Ç.; Writing Manuscript - C.Ö.Ç.; Critical Review - A.D.; Other - C.Ö.Ç., A.D.

Conflict of Interest: No conflict of interest was declared by the authors.

Financial Disclosure: The authors declared that this study has received no financial support.

• Editorial Notes
• Open access
• Published: 07 January 2020

What is useful research? The good, the bad, and the stable

• David M. Ozonoff 1 &
• Philippe Grandjean 2 , 3  

Environmental Health volume  19 , Article number:  2 ( 2020 ) Cite this article

12k Accesses

2 Citations

2 Altmetric

Metrics details

A scientific journal like Environmental Health strives to publish research that is useful within the field covered by the journal’s scope, in this case, public health. Useful research is more likely to make a difference. However, in many, if not most cases, the usefulness of an article can be difficult to ascertain until after its publication. Although replication is often thought of as a requirement for research to be considered valid, this criterion is retrospective and has resulted in a tendency toward inertia in environmental health research. An alternative viewpoint is that useful work is “stable”, i.e., not likely to be soon contradicted. We present this alternative view, which still relies on science being consensual, although pointing out that it is not the same as replicability, while not in contradiction. We believe that viewing potential usefulness of research reports through the lens of stability is a valuable perspective.

Good science as a purpose

Any scientific journal wishes to add to the general store of knowledge. For Environmental Health, an additional important goal is also to publish research that is useful for public health. While maximizing scientific validity is an irreducible minimum for any research journal, it does not guarantee that the outcome of a “good” article is useful. Most writing on this subject concerns efficiencies and criteria for generating new and useful research results while avoiding “research waste” [ 1 ]. In this regard, the role of journals is hard to define. Indeed, a usefulness objective depends upon what happens after publication, thus to some extent being out of our control. That said, because of the importance of this issue the Editors have set out to clarify our thinking about what makes published research useful.

First the obvious: properly conducted scientific research may not be useful, or worse, may potentially mislead, confuse or be erroneously interpreted. Journal editors and reviewers can mitigate such regrettable outcomes by being attentive to faulty over- or under-interpretation of properly generated data, and vice versa, ensuring that unrealistic standards don’t prevent publication of a “good” manuscript. In regard to the latter, we believe our journal should not shy away from alternative or novel interpretations that may be counter to established paradigms and have consciously adopted a precautionary orientation [ 2 ]: We believe that it is reasonable to feature risks that may seem remote at the moment because the history of environmental and occupational health is replete with instances of red flags ignored, resulting in horrific later harms that could no longer be mitigated [ 3 , 4 ].

Nonetheless, it has happened that researchers publishing results at odds with vested interests have become targets of unreasonable criticism and intimidation whose aim is to suppress or throw suspicion on unwelcome research information, as in the case of lead [ 3 , 5 ] and many other environmental chemicals [ 6 ]. An alternative counter strategy is generating new results favorable to a preferred view [ 7 , 8 ], with the objective of casting doubt on the uncomfortable research results. Indeed, one trade association involved in supporting such science once described its activities with the slogan, “Doubt is our product” [ 9 ]. Thus, for better or for worse, many people do not separate science, whether good or bad, from its implications [ 10 ].

Further, even without nefarious reasons, it is not uncommon for newly published research to be contradicted by additional results from other scientists. Not surprisingly, the public has become all too aware of findings whose apparent import is later found to be negligible, wrong, or cast into serious doubt, legitimately or otherwise [ 11 ]. This has been damaging to the discipline and its reputation [ 12 ].

Replication as a criterion

A principal reaction to this dilemma has been to demand that results be “replicated” before being put to use. As a result, both funding agencies [ 13 ] and journals [ 14 ] have announced their intention of emphasizing the reproducibility of research, thereby also facilitating replication [ 15 ]. On its face this sounds reasonable, but usual experimental or observational protocols are already based on internal replication. If some form of replication of a study is desired, attempts to duplicate an experimental set-up can easily produce non-identical measurements on repeated samples, and seemingly similar people in a population may yield somewhat different observations. Given an expected variability within and between studies, we need to define more precisely what is to be replicated and how it is to be judged.

That said, in most instances, it seems that what we are really asking for is interpretive replication (i.e., do we think two or more studies mean the same thing), not observational or measurement replication. Uninterpreted evidence is just raw data. The main product of scientific journals like Environmental Health is interpreted evidence. It is interpreted evidence that is actionable and likely to affect practice and policy.

Research stability

This brings us back to the question of what kind of evidence and its accompanying interpretation is likely to be of use? The philosopher Alex Broadbent distinguishes between how results get used and the decision about which results are likely to be used [ 16 ]. Discussions of research translation tend to focus on the former question, while the latter is rarely discussed. Broadbent introduces a new concept into the conversation, the stability of the research results.

He begins by identifying which results are not likely to be used. Broadbent observes that if a practitioner or policy-maker thinks a result might soon be overturned she is unlikely to use it. Since continual revision is a hallmark of science, this presents a dilemma. All results are open to revision as science progresses, so what users and policy makers really want are stable results, ones whose meaning is unlikely to change in ways that make a potential practice or policy quickly obsolete or wrong. What are the features of a stable result?

This is a trickier problem than it first appears. As Broadbent observes it does not seem sufficient to say that a stable a result is one that is not contradicted by subsequent work, an idea closely related to replication. Failure to contradict, like lack of replication, may have many reasons, including lack of interest, lack of funding, active suppression of research in a subject, or external events like social conflict or recession. Moreover, there are many examples of clinical practice, broadly accepted as stable in the non-contradiction sense, that have not been tested for one reason or another. Contrariwise, contradictory results may also be specious or fraudulent, e.g., due to attempts to make an unwelcome result appear unstable and hence unusable [ 6 , 9 ]. In sum, lack of contradiction doesn’t automatically make a result stable, nor does its presence annul the result.

One might plausibly think that the apparent truth of a scientific result would be sufficient to make a result stable. This is also in accordance with Naomi Oreskes’ emphasis of scientific knowledge being fundamentally consensual [ 10 ] and relies on the findings being generalizable [ 15 ]. Our journal, like most, employs conventional techniques like pre-publication peer review and editorial judgment, to maximize scientific validity of published articles; and we require Conflict of Interest declarations to maximize scientific integrity [ 6 , 17 ]. Still, a result may be true but not useful, and science that isn’t true may be very useful. Broadbent’s example of the latter is the most spectacular. Newtonian physics continues to be a paragon of usefulness despite the fact that in the age of Relativity Theory we know it to be false. Examples are also prevalent in environmental health. When John Snow identified contaminated water as a source of epidemic cholera in the mid-nineteenth Century he believed a toxin was the cause, as the germ theory of disease had not yet found purchase. This lack of understanding did not stop practitioners from advocating limiting exposure to sewage-contaminated water. Nonetheless, demands for modes of action or adverse outcome pathways are often used to block the use of new evidence on environmental hazards [ 18 ].

Criteria for stability

Broadbent’s suggestion is that a result likely to be seen as stable by practitioners and policy makers is one that (a) is not contradicted by good scientific evidence; and (b) would not likely be soon contradicted by further good research [ 16 ] (p. 63).

The first requirement, (a), simply says that any research that produces contradictory evidence be methodologically sound and free from bias, i.e., “good scientific evidence.” What constitutes “good” scientific evidence is a well discussed topic, of course, and not a novel requirement [ 1 ], but the stability frame puts existing quality criteria, in a different, perhaps more organized, structure, situating the evidence and its interpretation in relation to stability as a criterion for usefulness.

More novel is requirement (b), the belief that if further research were done it would not likely result in a contradiction. The if clause focuses our attention on examining instances where the indicated research has not yet been done. The criterion is therefore prospective, where the replication demand can only be used in retrospect.

This criterion could usefully be applied to inconclusive or underpowered studies that are often incorrectly labeled “negative” and interpreted to indicate “no risk” [ 18 ]. A U.S. National Research Council committee called attention to the erroneous inference that chemicals are regarded inert or safe, unless proven otherwise [ 19 ]. This “untested-chemical assumption” has resulted in exposure limits for only a small proportion of environmental chemicals, limits often later found to be much too high to adequately protect against adverse health effects [ 20 , 21 ]. For example, some current limits for perfluorinated compounds in drinking water do not protect against the immunotoxic effects in children and may be up to 100-fold too high [ 22 ].

Inertia as a consequence

Journals play an unfortunate part in the dearth of critical information on emerging contaminants, as published articles primarily address chemicals that have already been well studied [ 23 ]. This means that environmental health research suffers from an impoverishing inertia, which may in part be due to desired replications that may be superfluous or worse. The bottom line is that longstanding acceptance in the face of longstanding failure to test a proposition should not be used as a criterion of stability or of usefulness, although this is routinely done.

If non-contradiction, replication or truth are not reliable hallmarks of a potentially useful research result, then what is? Broadbent makes the tentative proposal that a stable interpretation is one which has a satisfactory answer to the question, “Why this interpretation rather than another?” Said another way, are there more likely, almost or equally as likely, or other possible explanations (including methodological error in the work in question)? Sometimes the answer is patently obvious. Such an evaluation is superfluous in instances where the outcomes have such forceful explanations that this exercise would be a waste of time, for example a construction worker falling from the staging. We only need one instance and (hopefully no repetitions) to make the case.

Consensus and stability

Having made the argument for perspicuous interpretation, we must also issue a note of caution. It is quite common to err in the other direction by downplaying conclusions and implications. Researchers frequently choose to hedge their conclusions by repeated use of words such as ‘maybe’, ‘perhaps’, ‘in theory’ and similar terms [ 24 ]. Indeed, we might call the hedge the official flower of epidemiology. To a policy maker, journalist or member of the public not familiar with the traditions of scientific writing, the caveats and reservations may sound like the new results are irredeemably tentative, leaving us with no justification for any intervention. To those with a vested interest, the soft wording can be exploited through selective quotation and by emphasizing real or alleged weaknesses [ 25 ]. This tendency goes beyond one’s own writings and affects peer review and evaluations of manuscripts and applications. Although skepticism is in the nature of science, a malignant form is the one that is veiled and expressed in terms of need for further replication or emphasizing limitations of otherwise stable observations [ 9 ]. By softening the conclusions and avoiding attribution of specific causality and the possible policy implications, researchers protect themselves against critique by appearing well-balanced, unassuming, or even skeptical toward one’s own findings. In seeking consensus, researchers often moderate or underestimate their findings, a tendency that is not in accordance with public health interests.

These are difficult issues, requiring a balancing act. The Editors continue to ponder the question how to inspire, improve and support the best research and its translation. We believe Broadbent’s stability idea is worth considering as an alternative perspective to the replication and research translation paradigms prevalent in discussions of this topic. We also believe in Oreskes’ vision of consensus, though not to a degree that will preclude new interpretations. Meanwhile, we will endeavor to keep the Journal’s standards high while encouraging work that will make a difference.

Availability of data and materials

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

Article   Google Scholar  

Grandjean P. Science for precautionary decision-making. In: Gee D, Grandjean P, Hansen SF, van den Hove S, MacGarvin M, Martin J, Nielsen G, Quist D, Stanners D, editors. Late Lessons from Early Warnings, vol. II. Copenhagen: European Environment Agency; 2013. p. 517–35.

Google Scholar  

Markowitz GE, Rosner D. Deceit and denial : the deadly politics of industrial pollution. Berkeley, CA: University of California Press; 2002.

European Environment Agency. Late lessons from early warnings: the precautionary principle 1896-2000. Environmental issue report No 22. Copenhagen: European Environment Agency; 2001.

Needleman HL. The removal of lead from gasoline: historical and personal reflections. Environ Res. 2000;84(1):20–35.

Article   CAS   Google Scholar  

Krimsky S. Science in the private interest. Lanham, MD: Rowman & Littlefield; 2003.

Boden LI, Ozonoff D. Litigation-generated science: why should we care? Environ Health Perspect. 2008;116(1):117–22.

Fabbri A, Lai A, Grundy Q, Bero LA. The influence of industry sponsorship on the research agenda: a scoping review. Am J Public Health. 2018;108(11):e9–e16.

Michaels D. Doubt is their product: how industry's assault on science threatens your health. Oxford; New York: Oxford University Press; 2008.

Oreskes N. Why trust science? Princeton, NJ: Princeton University Press; 2019.

Book   Google Scholar  

Nichols TM. The death of expertise : the campaign against established knowledge and why it matters. New York, NY: Oxford University Press; 2017.

Ioannidis JPA. All science should inform policy and regulation. PLoS Med. 2018;15(5):e1002576.

Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014;505(7485):612–3.

Journals unite for reproducibility. Nature. 2014;515(7525):7.

National Research Council. Reproducibility and replicability in science. Washington, DC: National Academies Press; 2019.

Broadbent A. Is stability a stable category in medical epistemology? Angewandte Philosophie. 2015;2(1):24–37.

Soskolne CL, Advani S, Sass J, Bero LA, Ruff K. Response to Acquavella J, conflict of interest: a hazard for epidemiology. Ann Epidemiol. 2019;36:62–3.

Grandjean P. Seven deadly sins of environmental epidemiology and the virtues of precaution. Epidemiology. 2008;19(1):158–62.

National Research Council. Science and decisions: advancing risk assessment. Washington, D.C: National Academy Press; 2009.

Goldsmith JR. Perspectives on what we formerly called threshold limit values. Am J Ind Med. 1991;19(6):805–12.

Schenk L, Hansson SO, Ruden C, Gilek M. Occupational exposure limits: a comparative study. Regul Toxicol Pharmacol. 2008;50(2):261–70.

Grandjean P, Budtz-Jorgensen E. Immunotoxicity of perfluorinated alkylates: calculation of benchmark doses based on serum concentrations in children. Environ Health. 2013;12:35.

Grandjean P, Eriksen ML, Ellegaard O, Wallin JA. The Matthew effect in environmental science publication: a bibliometric analysis of chemical substances in journal articles. Environ Health. 2011;10:96.

Hyland K. Hedging in scientific research articles. John Benjamins: Amsterdam; 1998.

Grandjean P. Late insights into early origins of disease. Basic Clin Pharmacol Toxicol. 2008;102(2):94–9.

Download references

Acknowledgements

Author information, authors and affiliations.

Department of Environmental Health, Boston University School of Public Health, Boston, MA, USA

David M. Ozonoff

Department of Environmental Medicine, University of Southern Denmark, Odense, Denmark

Philippe Grandjean

Department of Environmental Health, Harvard T.H. Chan School of Public Health, Boston, MA, USA

You can also search for this author in PubMed   Google Scholar

Contributions

DMO and PG jointly drafted the Editorial and read and approved the final version.

Corresponding author

Correspondence to Philippe Grandjean .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

DMO and PG are editors-in-chief of this journal but have no other interests to declare.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Ozonoff, D.M., Grandjean, P. What is useful research? The good, the bad, and the stable. Environ Health 19 , 2 (2020). https://doi.org/10.1186/s12940-019-0556-5

Download citation

Published : 07 January 2020

DOI : https://doi.org/10.1186/s12940-019-0556-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Generalizability
• Replication
• Reproducibility
• Scientific journals

Environmental Health

ISSN: 1476-069X

• General enquiries: [email protected]

Featured Topics

Featured series.

A series of random questions answered by Harvard experts.

Explore the Gazette

Read the latest.

How do you read organization’s silence over rise of Nazism?

Got milk? Does it give you problems?

Cancer risk, wine preference, and your genes

Good genes are nice, but joy is better.

Harvard Staff Writer

Harvard study, almost 80 years old, has proved that embracing community helps us live longer, and be happier

Part of the tackling issues of aging series.

A series on how Harvard researchers are tackling the problematic issues of aging.

W hen scientists began tracking the health of 268 Harvard sophomores in 1938 during the Great Depression, they hoped the longitudinal study would reveal clues to leading healthy and happy lives.

They got more than they wanted.

After following the surviving Crimson men for nearly 80 years as part of the Harvard Study of Adult Development , one of the world’s longest studies of adult life, researchers have collected a cornucopia of data on their physical and mental health.

Of the original Harvard cohort recruited as part of the Grant Study, only 19 are still alive, all in their mid-90s. Among the original recruits were eventual President John F. Kennedy and longtime Washington Post editor Ben Bradlee. (Women weren’t in the original study because the College was still all male.)

In addition, scientists eventually expanded their research to include the men’s offspring, who now number 1,300 and are in their 50s and 60s, to find out how early-life experiences affect health and aging over time. Some participants went on to become successful businessmen, doctors, lawyers, and others ended up as schizophrenics or alcoholics, but not on inevitable tracks.

“Loneliness kills. It’s as powerful as smoking or alcoholism.” Robert Waldinger, psychiatrist, Massachusetts General Hospital

During the intervening decades, the control groups have expanded. In the 1970s, 456 Boston inner-city residents were enlisted as part of the Glueck Study, and 40 of them are still alive. More than a decade ago, researchers began including wives in the Grant and Glueck studies.

Over the years, researchers have studied the participants’ health trajectories and their broader lives, including their triumphs and failures in careers and marriage, and the finding have produced startling lessons, and not only for the researchers.

“The surprising finding is that our relationships and how happy we are in our relationships has a powerful influence on our health,” said Robert Waldinger , director of the study, a psychiatrist at Massachusetts General Hospital and a professor of psychiatry at Harvard Medical School . “Taking care of your body is important, but tending to your relationships is a form of self-care too. That, I think, is the revelation.”

Close relationships, more than money or fame, are what keep people happy throughout their lives, the study revealed. Those ties protect people from life’s discontents, help to delay mental and physical decline, and are better predictors of long and happy lives than social class, IQ, or even genes. That finding proved true across the board among both the Harvard men and the inner-city participants.

“The people who were the most satisfied in their relationships at age 50 were the healthiest at age 80,” said Robert Waldinger with his wife Jennifer Stone.

Rose Lincoln/Harvard Staff Photographer

The long-term research has received funding from private foundations, but has been financed largely by grants from the National Institutes of Health, first through the National Institute of Mental Health, and more recently through the National Institute on Aging.

Researchers who have pored through data, including vast medical records and hundreds of in-person interviews and questionnaires, found a strong correlation between men’s flourishing lives and their relationships with family, friends, and community. Several studies found that people’s level of satisfaction with their relationships at age 50 was a better predictor of physical health than their cholesterol levels were.

“When we gathered together everything we knew about them about at age 50, it wasn’t their middle-age cholesterol levels that predicted how they were going to grow old,” said Waldinger in a popular TED Talk . “It was how satisfied they were in their relationships. The people who were the most satisfied in their relationships at age 50 were the healthiest at age 80.”

He recorded his TED talk, titled “What Makes a Good Life? Lessons from the Longest Study on Happiness,” in 2015, and it has been viewed 13,000,000 times.

The researchers also found that marital satisfaction has a protective effect on people’s mental health. Part of a study found that people who had happy marriages in their 80s reported that their moods didn’t suffer even on the days when they had more physical pain. Those who had unhappy marriages felt both more emotional and physical pain.

Those who kept warm relationships got to live longer and happier, said Waldinger, and the loners often died earlier. “Loneliness kills,” he said. “It’s as powerful as smoking or alcoholism.”

According to the study, those who lived longer and enjoyed sound health avoided smoking and alcohol in excess. Researchers also found that those with strong social support experienced less mental deterioration as they aged.

In part of a recent study , researchers found that women who felt securely attached to their partners were less depressed and more happy in their relationships two-and-a-half years later, and also had better memory functions than those with frequent marital conflicts.

“When the study began, nobody cared about empathy or attachment. But the key to healthy aging is relationships, relationships, relationships.” George Vaillant, psychiatrist

“Good relationships don’t just protect our bodies; they protect our brains,” said Waldinger in his TED talk. “And those good relationships, they don’t have to be smooth all the time. Some of our octogenarian couples could bicker with each other day in and day out, but as long as they felt that they could really count on the other when the going got tough, those arguments didn’t take a toll on their memories.”

Since aging starts at birth, people should start taking care of themselves at every stage of life, the researchers say.

“Aging is a continuous process,” Waldinger said. “You can see how people can start to differ in their health trajectory in their 30s, so that by taking good care of yourself early in life you can set yourself on a better course for aging. The best advice I can give is ‘Take care of your body as though you were going to need it for 100 years,’ because you might.”

The study, like its remaining original subjects, has had a long life, spanning four directors, whose tenures reflected their medical interests and views of the time.

Under the first director, Clark Heath, who stayed from 1938 until 1954, the study mirrored the era’s dominant view of genetics and biological determinism. Early researchers believed that physical constitution, intellectual ability, and personality traits determined adult development. They made detailed anthropometric measurements of skulls, brow bridges, and moles, wrote in-depth notes on the functioning of major organs, examined brain activity through electroencephalograms, and even analyzed the men’s handwriting.

Now, researchers draw men’s blood for DNA testing and put them into MRI scanners to examine organs and tissues in their bodies, procedures that would have sounded like science fiction back in 1938. In that sense, the study itself represents a history of the changes that life brings.

6 factors predicting healthy aging According to George Vaillant’s book “Aging Well,” from observations of Harvard men in long-term aging study

Physically active.

Absence of alcohol abuse and smoking

Having mature mechanisms to cope with life’s ups and downs

Healthy weight

Stable marriage.

Psychiatrist George Vaillant, who joined the team as a researcher in 1966, led the study from 1972 until 2004. Trained as a psychoanalyst, Vaillant emphasized the role of relationships, and came to recognize the crucial role they played in people living long and pleasant lives.

In a book called “Aging Well,” Vaillant wrote that six factors predicted healthy aging for the Harvard men: physical activity, absence of alcohol abuse and smoking, having mature mechanisms to cope with life’s ups and downs, and enjoying both a healthy weight and a stable marriage. For the inner-city men, education was an additional factor. “The more education the inner city men obtained,” wrote Vaillant, “the more likely they were to stop smoking, eat sensibly, and use alcohol in moderation.”

Vaillant’s research highlighted the role of these protective factors in healthy aging. The more factors the subjects had in place, the better the odds they had for longer, happier lives.

“When the study began, nobody cared about empathy or attachment,” said Vaillant. “But the key to healthy aging is relationships, relationships, relationships.”

“We want to find out how it is that a difficult childhood reaches across decades to break down the body in middle age and later.” Robert Waldinger

The study showed that the role of genetics and long-lived ancestors proved less important to longevity than the level of satisfaction with relationships in midlife, now recognized as a good predictor of healthy aging. The research also debunked the idea that people’s personalities “set like plaster” by age 30 and cannot be changed.

“Those who were clearly train wrecks when they were in their 20s or 25s turned out to be wonderful octogenarians,” he said. “On the other hand, alcoholism and major depression could take people who started life as stars and leave them at the end of their lives as train wrecks.”

The study’s fourth director, Waldinger has expanded research to the wives and children of the original men. That is the second-generation study, and Waldinger hopes to expand it into the third and fourth generations. “It will probably never be replicated,” he said of the lengthy research, adding that there is yet more to learn.

“We’re trying to see how people manage stress, whether their bodies are in a sort of chronic ‘fight or flight’ mode,” Waldinger said. “We want to find out how it is that a difficult childhood reaches across decades to break down the body in middle age and later.”

Lara Tang ’18, a human and evolutionary biology concentrator who recently joined the team as a research assistant, relishes the opportunity to help find some of those answers. She joined the effort after coming across Waldinger’s TED talk in one of her classes.

“That motivated me to do more research on adult development,” said Tang. “I want to see how childhood experiences affect developments of physical health, mental health, and happiness later in life.”

Asked what lessons he has learned from the study, Waldinger, who is a Zen priest, said he practices meditation daily and invests time and energy in his relationships, more than before.

“It’s easy to get isolated, to get caught up in work and not remembering, ‘Oh, I haven’t seen these friends in a long time,’ ” Waldinger said. “So I try to pay more attention to my relationships than I used to.”

Share this article

Also in this series:.

Probe of Alzheimer’s follows paths of infection

Starting with microbes, Harvard-MGH researchers outline a devastating chain of events

To age better, eat better

Much of life is beyond our control, but dining smartly can help us live healthier, longer

The balance in healthy aging

To grow old well requires minimizing accidents, such as falling, as well as ailments

How old can we get? It might be written in stem cells

No clock, no crystal ball, but lots of excitement — and ambition — among Harvard scientists

Plotting the demise of Alzheimer’s

New study is major test for power of early action

You might like

Medical historians look to cultural context, work of peer publications in wrestling with case of New England Journal of Medicine

Biomolecular archaeologist looks at why most of world’s population has trouble digesting beverage that helped shape civilization

Biologist separates reality of science from the claims of profiling firms

Epic science inside a cubic millimeter of brain

Researchers publish largest-ever dataset of neural connections

Pop star on one continent, college student on another

Model and musician Kazuma Mitchell managed to (mostly) avoid the spotlight while at Harvard

Finding right mix on campus speech policies

Legal, political scholars discuss balancing personal safety, constitutional rights, academic freedom amid roiling protests, cultural shifts