critically evaluate research paper

Write a Critical Review of a Scientific Journal Article

1. identify how and why the research was carried out, 2. establish the research context, 3. evaluate the research, 4. establish the significance of the research.

• Writing Your Critique

Ask Us: Chat, email, visit or call

Video: How to Integrate Critical Voice into Your Literature Review

Video: Note-taking and Writing Tips to Avoid Plagiarism

Get assistance

The library offers a range of helpful services.  All of our appointments are free of charge and confidential.

• Book an appointment

Read the article(s) carefully and use the questions below to help you identify how and why the research was carried out. Look at the following sections: 

Introduction

• What was the objective of the study?
• What methods were used to accomplish this purpose (e.g., systematic recording of observations, analysis and evaluation of published research, assessment of theory, etc.)?
• What techniques were used and how was each technique performed?
• What kind of data can be obtained using each technique?
• How are such data interpreted?
• What kind of information is produced by using the technique?
• What objective evidence was obtained from the authors’ efforts (observations, measurements, etc.)?
• What were the results of the study? 
• How was each technique used to obtain each result?
• What statistical tests were used to evaluate the significance of the conclusions based on numeric or graphic data?
• How did each result contribute to answering the question or testing the hypothesis raised in the introduction?
• How were the results interpreted? How were they related to the original problem (authors’ view of evidence rather than objective findings)? 
• Were the authors able to answer the question (test the hypothesis) raised?
• Did the research provide new factual information, a new understanding of a phenomenon in the field, or a new research technique?
• How was the significance of the work described?
• Do the authors relate the findings of the study to literature in the field?
• Did the reported observations or interpretations support or refute observations or interpretations made by other researchers?

These questions were adapted from the following sources:  Kuyper, B.J. (1991). Bringing up scientists in the art of critiquing research. Bioscience 41(4), 248-250. Wood, J.M. (2003). Research Lab Guide. MICR*3260 Microbial Adaptation and Development Web Site . Retrieved July 31, 2006.

Once you are familiar with the article, you can establish the research context by asking the following questions:

• Who conducted the research? What were/are their interests?
• When and where was the research conducted?
• Why did the authors do this research?
• Was this research pertinent only within the authors’ geographic locale, or did it have broader (even global) relevance?
• Were many other laboratories pursuing related research when the reported work was done? If so, why?
• For experimental research, what funding sources met the costs of the research?
• On what prior observations was the research based? What was and was not known at the time?
• How important was the research question posed by the researchers?

These questions were adapted from the following sources: Kuyper, B.J. (1991). Bringing up scientists in the art of critiquing research. Bioscience 41(4), 248-250. Wood, J.M. (2003). Research Lab Guide. MICR*3260 Microbial Adaptation and Development Web Site . Retrieved July 31, 2006.

Remember that simply disagreeing with the material is not considered to be a critical assessment of the material.  For example, stating that the sample size is insufficient is not a critical assessment.  Describing why the sample size is insufficient for the claims being made in the study would be a critical assessment.

Use the questions below to help you evaluate the quality of the authors’ research:

• Does the title precisely state the subject of the paper?
• Read the statement of purpose in the abstract. Does it match the one in the introduction?

Acknowledgments

• Could the source of the research funding have influenced the research topic or conclusions?
• Check the sequence of statements in the introduction. Does all the information lead coherently to the purpose of the study?
• Review all methods in relation to the objective(s) of the study. Are the methods valid for studying the problem?
• Check the methods for essential information. Could the study be duplicated from the methods and information given?
• Check the methods for flaws. Is the sample selection adequate? Is the experimental design sound?
• Check the sequence of statements in the methods. Does all the information belong there? Is the sequence of methods clear and pertinent?
• Was there mention of ethics? Which research ethics board approved the study?
• Carefully examine the data presented in the tables and diagrams. Does the title or legend accurately describe the content? 
• Are column headings and labels accurate? 
• Are the data organized for ready comparison and interpretation? (A table should be self-explanatory, with a title that accurately and concisely describes content and column headings that accurately describe information in the cells.)
• Review the results as presented in the text while referring to the data in the tables and diagrams. Does the text complement, and not simply repeat data? Are there discrepancies between the results in the text and those in the tables?
• Check all calculations and presentation of data.
• Review the results in light of the stated objectives. Does the study reveal what the researchers intended?
• Does the discussion clearly address the objectives and hypotheses?
• Check the interpretation against the results. Does the discussion merely repeat the results? 
• Does the interpretation arise logically from the data or is it too far-fetched? 
• Have the faults, flaws, or shortcomings of the research been addressed?
• Is the interpretation supported by other research cited in the study?
• Does the study consider key studies in the field?
• What is the significance of the research? Do the authors mention wider implications of the findings?
• Is there a section on recommendations for future research? Are there other research possibilities or directions suggested? 

Consider the article as a whole

• Reread the abstract. Does it accurately summarize the article?
• Check the structure of the article (first headings and then paragraphing). Is all the material organized under the appropriate headings? Are sections divided logically into subsections or paragraphs?
• Are stylistic concerns, logic, clarity, and economy of expression addressed?

These questions were adapted from the following sources:  Kuyper, B.J. (1991). Bringing up scientists in the art of critiquing research. Bioscience 41(4), 248-250. Wood, J.M. (2003). Research Lab Guide. MICR*3260 Microbial Adaptation and Development Web Site. Retrieved July 31, 2006.

After you have evaluated the research, consider whether the research has been successful. Has it led to new questions being asked, or new ways of using existing knowledge? Are other researchers citing this paper?

You should consider the following questions:

• How did other researchers view the significance of the research reported by your authors?
• Did the research reported in your article result in the formulation of new questions or hypotheses (by the authors or by other researchers)?
• Have other researchers subsequently supported or refuted the observations or interpretations of these authors?
• Did the research make a significant contribution to human knowledge?
• Did the research produce any practical applications?
• What are the social, political, technological, medical implications of this research?
• How do you evaluate the significance of the research?

To answer these questions, look at review articles to find out how reviewers view this piece of research. Look at research articles and databases like Web of Science to see how other people have used this work. What range of journals have cited this article?

These questions were adapted from the following sources:

Kuyper, B.J. (1991). Bringing up scientists in the art of critiquing research. Bioscience 41(4), 248-250. Wood, J.M. (2003). Research Lab Guide. MICR*3260 Microbial Adaptation and Development Web Site . Retrieved July 31, 2006.

• << Previous: Start Here
• Next: Writing Your Critique >>
• Last Updated: Jan 11, 2024 12:42 PM
• URL: https://guides.lib.uoguelph.ca/WriteCriticalReview

Suggest an edit to this guide

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

• The Open University
• Accessibility hub
• Guest user / Sign out
• Study with The Open University

My OpenLearn Profile

Personalise your OpenLearn profile, save your favourite content and get recognition for your learning

About this free course

Become an ou student, download this course, share this free course.

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

1 Important points to consider when critically evaluating published research papers

Simple review articles (also referred to as ‘narrative’ or ‘selective’ reviews), systematic reviews and meta-analyses provide rapid overviews and ‘snapshots’ of progress made within a field, summarising a given topic or research area. They can serve as useful guides, or as current and comprehensive ‘sources’ of information, and can act as a point of reference to relevant primary research studies within a given scientific area. Narrative or systematic reviews are often used as a first step towards a more detailed investigation of a topic or a specific enquiry (a hypothesis or research question), or to establish critical awareness of a rapidly-moving field (you will be required to demonstrate this as part of an assignment, an essay or a dissertation at postgraduate level).

The majority of primary ‘empirical’ research papers essentially follow the same structure (abbreviated here as IMRAD). There is a section on Introduction, followed by the Methods, then the Results, which includes figures and tables showing data described in the paper, and a Discussion. The paper typically ends with a Conclusion, and References and Acknowledgements sections.

The Title of the paper provides a concise first impression. The Abstract follows the basic structure of the extended article. It provides an ‘accessible’ and concise summary of the aims, methods, results and conclusions. The Introduction provides useful background information and context, and typically outlines the aims and objectives of the study. The Abstract can serve as a useful summary of the paper, presenting the purpose, scope and major findings. However, simply reading the abstract alone is not a substitute for critically reading the whole article. To really get a good understanding and to be able to critically evaluate a research study, it is necessary to read on.

While most research papers follow the above format, variations do exist. For example, the results and discussion sections may be combined. In some journals the materials and methods may follow the discussion, and in two of the most widely read journals, Science and Nature, the format does vary from the above due to restrictions on the length of articles. In addition, there may be supporting documents that accompany a paper, including supplementary materials such as supporting data, tables, figures, videos and so on. There may also be commentaries or editorials associated with a topical research paper, which provide an overview or critique of the study being presented.

Box 1 Key questions to ask when appraising a research paper

• Is the study’s research question relevant?
• Does the study add anything new to current knowledge and understanding?
• Does the study test a stated hypothesis?
• Is the design of the study appropriate to the research question?
• Do the study methods address key potential sources of bias?
• Were suitable ‘controls’ included in the study?
• Were the statistical analyses appropriate and applied correctly?
• Is there a clear statement of findings?
• Does the data support the authors’ conclusions?
• Are there any conflicts of interest or ethical concerns?

There are various strategies used in reading a scientific research paper, and one of these is to start with the title and the abstract, then look at the figures and tables, and move on to the introduction, before turning to the results and discussion, and finally, interrogating the methods.

Another strategy (outlined below) is to begin with the abstract and then the discussion, take a look at the methods, and then the results section (including any relevant tables and figures), before moving on to look more closely at the discussion and, finally, the conclusion. You should choose a strategy that works best for you. However, asking the ‘right’ questions is a central feature of critical appraisal, as with any enquiry, so where should you begin? Here are some critical questions to consider when evaluating a research paper.

Look at the Abstract and then the Discussion : Are these accessible and of general relevance or are they detailed, with far-reaching conclusions? Is it clear why the study was undertaken? Why are the conclusions important? Does the study add anything new to current knowledge and understanding? The reasons why a particular study design or statistical method were chosen should also be clear from reading a research paper. What is the research question being asked? Does the study test a stated hypothesis? Is the design of the study appropriate to the research question? Have the authors considered the limitations of their study and have they discussed these in context?

Take a look at the Methods : Were there any practical difficulties that could have compromised the study or its implementation? Were these considered in the protocol? Were there any missing values and, if so, was the number of missing values too large to permit meaningful analysis? Was the number of samples (cases or participants) too small to establish meaningful significance? Do the study methods address key potential sources of bias? Were suitable ‘controls’ included in the study? If controls are missing or not appropriate to the study design, we cannot be confident that the results really show what is happening in an experiment. Were the statistical analyses appropriate and applied correctly? Do the authors point out the limitations of methods or tests used? Were the methods referenced and described in sufficient detail for others to repeat or extend the study?

Take a look at the Results section and relevant tables and figures : Is there a clear statement of findings? Were the results expected? Do they make sense? What data supports them? Do the tables and figures clearly describe the data (highlighting trends etc.)? Try to distinguish between what the data show and what the authors say they show (i.e. their interpretation).

Moving on to look in greater depth at the Discussion and Conclusion : Are the results discussed in relation to similar (previous) studies? Do the authors indulge in excessive speculation? Are limitations of the study adequately addressed? Were the objectives of the study met and the hypothesis supported or refuted (and is a clear explanation provided)? Does the data support the authors’ conclusions? Maybe there is only one experiment to support a point. More often, several different experiments or approaches combine to support a particular conclusion. A rule of thumb here is that if multiple approaches and multiple lines of evidence from different directions are presented, and all point to the same conclusion, then the conclusions are more credible. But do question all assumptions. Identify any implicit or hidden assumptions that the authors may have used when interpreting their data. Be wary of data that is mixed up with interpretation and speculation! Remember, just because it is published, does not mean that it is right.

O ther points you should consider when evaluating a research paper : Are there any financial, ethical or other conflicts of interest associated with the study, its authors and sponsors? Are there ethical concerns with the study itself? Looking at the references, consider if the authors have preferentially cited their own previous publications (i.e. needlessly), and whether the list of references are recent (ensuring that the analysis is up-to-date). Finally, from a practical perspective, you should move beyond the text of a research paper, talk to your peers about it, consult available commentaries, online links to references and other external sources to help clarify any aspects you don’t understand.

The above can be taken as a general guide to help you begin to critically evaluate a scientific research paper, but only in the broadest sense. Do bear in mind that the way that research evidence is critiqued will also differ slightly according to the type of study being appraised, whether observational or experimental, and each study will have additional aspects that would need to be evaluated separately. For criteria recommended for the evaluation of qualitative research papers, see the article by Mildred Blaxter (1996), available online. Details are in the References.

Activity 1 Critical appraisal of a scientific research paper

A critical appraisal checklist, which you can download via the link below, can act as a useful tool to help you to interrogate research papers. The checklist is divided into four sections, broadly covering:

• some general aspects
• research design and methodology
• the results
• discussion, conclusion and references.

Science perspective – critical appraisal checklist [ Tip: hold Ctrl and click a link to open it in a new tab. ( Hide tip ) ]

• Identify and obtain a research article based on a topic of your own choosing, using a search engine such as Google Scholar or PubMed (for example).
• The selection criteria for your target paper are as follows: the article must be an open access primary research paper (not a review) containing empirical data, published in the last 2–3 years, and preferably no more than 5–6 pages in length.
• Critically evaluate the research paper using the checklist provided, making notes on the key points and your overall impression.

Critical appraisal checklists are useful tools to help assess the quality of a study. Assessment of various factors, including the importance of the research question, the design and methodology of a study, the validity of the results and their usefulness (application or relevance), the legitimacy of the conclusions, and any potential conflicts of interest, are an important part of the critical appraisal process. Limitations and further improvements can then be considered.

Evaluating Research Articles

Understanding research statistics, critical appraisal, help us improve the libguide.

Imagine for a moment that you are trying to answer a clinical (PICO) question regarding one of your patients/clients. Do you know how to determine if a research study is of high quality? Can you tell if it is applicable to your question? In evidence based practice, there are many things to look for in an article that will reveal its quality and relevance. This guide is a collection of resources and activities that will help you learn how to evaluate articles efficiently and accurately.

Is health research new to you? Or perhaps you're a little out of practice with reading it? The following questions will help illuminate an article's strengths or shortcomings. Ask them of yourself as you are reading an article:

• Is the article peer reviewed?
• Are there any conflicts of interest based on the author's affiliation or the funding source of the research?
• Are the research questions or objectives clearly defined?
• Is the study a systematic review or meta analysis?
• Is the study design appropriate for the research question?
• Is the sample size justified? Do the authors explain how it is representative of the wider population?
• Do the researchers describe the setting of data collection?
• Does the paper clearly describe the measurements used?
• Did the researchers use appropriate statistical measures?
• Are the research questions or objectives answered?
• Did the researchers account for confounding factors?
• Have the researchers only drawn conclusions about the groups represented in the research?
• Have the authors declared any conflicts of interest?

If the answer to these questions about an article you are reading are mostly YESes , then it's likely that the article is of decent quality. If the answers are most NOs , then it may be a good idea to move on to another article. If the YESes and NOs are roughly even, you'll have to decide for yourself if the article is good enough quality for you. Some factors, like a poor literature review, are not as important as the researchers neglecting to describe the measurements they used. As you read more research, you'll be able to more easily identify research that is well done vs. that which is not well done.

Determining if a research study has used appropriate statistical measures is one of the most critical and difficult steps in evaluating an article. The following links are great, quick resources for helping to better understand how to use statistics in health research.

• How to read a paper: Statistics for the non-statistician. II: “Significant” relations and their pitfalls This article continues the checklist of questions that will help you to appraise the statistical validity of a paper. Greenhalgh Trisha. How to read a paper: Statistics for the non-statistician. II: “Significant” relations and their pitfalls BMJ 1997; 315 :422 *On the PMC PDF, you need to scroll past the first article to get to this one.*
• A consumer's guide to subgroup analysis The extent to which a clinician should believe and act on the results of subgroup analyses of data from randomized trials or meta-analyses is controversial. Guidelines are provided in this paper for making these decisions.

Statistical Versus Clinical Significance

When appraising studies, it's important to consider both the clinical and statistical significance of the research. This video offers a quick explanation of why.

If you have a little more time, this video explores statistical and clinical significance in more detail, including examples of how to calculate an effect size.

• Statistical vs. Clinical Significance Transcript Transcript document for the Statistical vs. Clinical Significance video.
• Effect Size Transcript Transcript document for the Effect Size video.
• P Values, Statistical Significance & Clinical Significance This handout also explains clinical and statistical significance.
• Absolute versus relative risk – making sense of media stories Understanding the difference between relative and absolute risk is essential to understanding statistical tests commonly found in research articles.

Critical appraisal is the process of systematically evaluating research using established and transparent methods. In critical appraisal, health professionals use validated checklists/worksheets as tools to guide their assessment of the research. It is a more advanced way of evaluating research than the more basic method explained above. To learn more about critical appraisal or to access critical appraisal tools, visit the websites below.

• Last Updated: Jun 11, 2024 10:26 AM
• URL: https://libguides.massgeneral.org/evaluatingarticles

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Indian Prosthodont Soc
• v.21(1); Jan-Mar 2021

Critical evaluation of publications

N. gopi chander.

Editor, The Journal of Indian Prosthodontic Society, Chennai, Tamil Nadu, India

Critical evaluation is the process of examining the research for the strength or weakness of the findings, validity, relevance, and usefulness of the research findings.[ 1 ] The availability of extensive information and the difficulty in differentiating the relevant information obligate the primary need of critical appraisal. In addition, it establishes superior evidence and increases the application to clinical practice.[ 2 ] More importantly, it differentiates between significant and/or insignificant data in the literature and aids in providing the updated information. The purpose of critical appraisal shall help in informed decision and improve the quality of healthcare provided to patients.[ 1 , 2 , 3 ]

The research data have three possible outcomes – true findings, random variation that occurs due to chance, and biased results due to systematic error.[ 4 ] The true findings can be of positive or negative results, but it shall be highly recognized. The random error or actual result deviation occurs due to the uncontrollable factors such as smaller sample size and confounding factors. The random error does not alter the measured value, but it is an imperfect error caused due to study design inconsistencies. These errors are unpredictable and cannot be repeated again by repeating the analysis. The biased results are deliberate deviation in the study design, methodology, or investigations. The deviations in the result can be due to poor designing, to the methodology, or in the analysis. It will be difficult to differentiate these findings without critical analysis of the literature.[ 5 , 6 ]

There are various guidelines and tools proposed to critically evaluate the literature.[ 7 , 8 , 9 ] Since the scientific literature is in constant evolution, no one guidelines or checklist is considered to be gold standard. Moreover, the appraisal varies with the type of research. The checklist provided by various organizations for designing or structuring manuscripts - case report, reviews, and original research - cannot be combined or generalized for use. Similarly, it varies with the types of study design - randomized clinical trials and observational studies –case–control, cohort, and cross-sectional studies. The methodological guidelines such as consort statements, CARE guidelines, PROSPERO, or Cochrane checklists can significantly aid in the evaluation of different types of research data.[ 10 ] The structured approach and checklists provided by the organizations can be a valuable aid to conduct research as well as critically evaluate the manuscripts. In addition to the guidelines, the simplified checklists proposed by Young and Solomon can be of adjuvant tool in critical assessment of the literature.[ 1 ] It consists of 10 simple rules. That includes relevance of study question, new information to existing literature, type of research question, appropriateness of study design, bias appraisal, adherence of study protocol, hypothesis testing, check or estimation of statistical analysis, validation of conclusion, and identification of conflicts of interest. These checklists along with updated methodological guidelines for different types of study designs can be a valuable tool for critical appraisal of the literature.[ 1 , 10 ]

Most of the tools assess the validity, reliability, bias, and clinical application of the research data. The validity aids in determining the accuracy of the results, and the reliability establishes the consistency of the results. The bias is systemic deviation of results. The bias is of many types: it can be of from the initiation of the study to manuscript publication. Various assessment tools have been proposed to determine the bias. More commonly employed are the GRADE, Grade pro, Newcastle Ottawa, jaded, ROB 2, and ARRIVE 2.[ 11 ] The bias tools vary with the type of study design, and it is significant to use the appropriate tool. The tools assess and grade the quality of bias in the manuscript. These tools are majorly used for evaluating randomized control trial employed for systematic review and meta-analysis but can be suitably employed to different study designs. These tools provide the grading of bias and provide useful data that are essential for clinical application.[ 11 , 12 ]

Rapid appraisal can be done with merit trials/rapid critical appraisal tool.[ 6 ] It is a compressed tool that basically assesses on the validity, reliability, and clinical use of the study. This is a simplified checklist for quicker assessment; however, for more accurate assessment, it is essential to appraise the entire manuscript from introduction till the conclusion. This mandates a detailed check for every component of the literature in accordance to the standard guidelines. In addition, the journal indexing and metrics can play a significant role in estimation. Higher metric journal shall possess more rigorous peer-review process that reduces the significant errors in the manuscript.[ 3 , 4 ]

The major contents to be generally assessed in the introduction of the manuscript are type and contents of research question, justification of purpose/background of the study with articles published in the last 5 years, or older articles that possess significant influences, citations of peer-reviewed journal, defined objective, and hypothesis statement. In methodology, the parameter of appraisal parameters should be on study design, inclusion and exclusion criteria, care in reduction of bias, following the acceptable procedures, control on confounding variables, and valid outcome measures. The result section should be checked for the subject and baseline - demographic, relevant statistical tests, and statistical significance. The discussion should possess adequate literature substantiation for results, study limitations, and declarations on conflicts of interest.[ 6 ]

In the prosthodontic literature, extensive reports of similar nature exist; critical analysis of the literature is a necessary skill to be mastered by researchers and clinicians.[ 10 ] It helps clinicians to make quality evidenced healthcare decisions by extensive evaluation of the literature.

We’re reviewing our resources this spring (May-August 2024). We will do our best to minimize disruption, but you might notice changes over the next few months as we correct errors & delete redundant resources. 

Critical Analysis and Evaluation

Many assignments ask you to   critique   and   evaluate   a source. Sources might include journal articles, books, websites, government documents, portfolios, podcasts, or presentations.

When you   critique,   you offer both negative and positive analysis of the content, writing, and structure of a source.

When   you   evaluate , you assess how successful a source is at presenting information, measured against a standard or certain criteria.

Elements of a critical analysis:

opinion + evidence from the article + justification

Your   opinion   is your thoughtful reaction to the piece.

Evidence from the article  offers some proof to back up your opinion.

The   justification   is an explanation of how you arrived at your opinion or why you think it’s true.

How do you critique and evaluate?

When critiquing and evaluating someone else’s writing/research, your purpose is to reach an   informed opinion   about a source. In order to do that, try these three steps:

• How do you feel?
• What surprised you?
• What left you confused?
• What pleased or annoyed you?
• What was interesting?
• What is the purpose of this text?
• Who is the intended audience?
• What kind of bias is there?
• What was missing?
• See our resource on analysis and synthesis ( Move From Research to Writing: How to Think ) for other examples of questions to ask.
• sophisticated
• interesting
• undocumented
• disorganized
• superficial
• unconventional
• inappropriate interpretation of evidence
• unsound or discredited methodology
• traditional
• unsubstantiated
• unsupported
• well-researched
• easy to understand
• Opinion : This article’s assessment of the power balance in cities is   confusing.
• Evidence:   It first says that the power to shape policy is evenly distributed among citizens, local government, and business (Rajal, 232).
• Justification :  but then it goes on to focus almost exclusively on business. Next, in a much shorter section, it combines the idea of citizens and local government into a single point of evidence. This leaves the reader with the impression that the citizens have no voice at all. It is   not helpful   in trying to determine the role of the common voter in shaping public policy.  

Sample criteria for critical analysis

Sometimes the assignment will specify what criteria to use when critiquing and evaluating a source. If not, consider the following prompts to approach your analysis. Choose the questions that are most suitable for your source.

• What do you think about the quality of the research? Is it significant?
• Did the author answer the question they set out to? Did the author prove their thesis?
• Did you find contradictions to other things you know?
• What new insight or connections did the author make?
• How does this piece fit within the context of your course, or the larger body of research in the field?
• The structure of an article or book is often dictated by standards of the discipline or a theoretical model. Did the piece meet those standards?
• Did the piece meet the needs of the intended audience?
• Was the material presented in an organized and logical fashion?
• Is the argument cohesive and convincing? Is the reasoning sound? Is there enough evidence?
• Is it easy to read? Is it clear and easy to understand, even if the concepts are sophisticated?

• Privacy Policy

Home » Evaluating Research – Process, Examples and Methods

Evaluating Research – Process, Examples and Methods

Table of Contents

Evaluating Research

Definition:

Evaluating Research refers to the process of assessing the quality, credibility, and relevance of a research study or project. This involves examining the methods, data, and results of the research in order to determine its validity, reliability, and usefulness. Evaluating research can be done by both experts and non-experts in the field, and involves critical thinking, analysis, and interpretation of the research findings.

Research Evaluating Process

The process of evaluating research typically involves the following steps:

Identify the Research Question

The first step in evaluating research is to identify the research question or problem that the study is addressing. This will help you to determine whether the study is relevant to your needs.

Assess the Study Design

The study design refers to the methodology used to conduct the research. You should assess whether the study design is appropriate for the research question and whether it is likely to produce reliable and valid results.

Evaluate the Sample

The sample refers to the group of participants or subjects who are included in the study. You should evaluate whether the sample size is adequate and whether the participants are representative of the population under study.

Review the Data Collection Methods

You should review the data collection methods used in the study to ensure that they are valid and reliable. This includes assessing the measures used to collect data and the procedures used to collect data.

Examine the Statistical Analysis

Statistical analysis refers to the methods used to analyze the data. You should examine whether the statistical analysis is appropriate for the research question and whether it is likely to produce valid and reliable results.

Assess the Conclusions

You should evaluate whether the data support the conclusions drawn from the study and whether they are relevant to the research question.

Consider the Limitations

Finally, you should consider the limitations of the study, including any potential biases or confounding factors that may have influenced the results.

Evaluating Research Methods

Evaluating Research Methods are as follows:

• Peer review: Peer review is a process where experts in the field review a study before it is published. This helps ensure that the study is accurate, valid, and relevant to the field.
• Critical appraisal : Critical appraisal involves systematically evaluating a study based on specific criteria. This helps assess the quality of the study and the reliability of the findings.
• Replication : Replication involves repeating a study to test the validity and reliability of the findings. This can help identify any errors or biases in the original study.
• Meta-analysis : Meta-analysis is a statistical method that combines the results of multiple studies to provide a more comprehensive understanding of a particular topic. This can help identify patterns or inconsistencies across studies.
• Consultation with experts : Consulting with experts in the field can provide valuable insights into the quality and relevance of a study. Experts can also help identify potential limitations or biases in the study.
• Review of funding sources: Examining the funding sources of a study can help identify any potential conflicts of interest or biases that may have influenced the study design or interpretation of results.

Example of Evaluating Research

Example of Evaluating Research sample for students:

Title of the Study: The Effects of Social Media Use on Mental Health among College Students

Sample Size: 500 college students

Sampling Technique : Convenience sampling

• Sample Size: The sample size of 500 college students is a moderate sample size, which could be considered representative of the college student population. However, it would be more representative if the sample size was larger, or if a random sampling technique was used.
• Sampling Technique : Convenience sampling is a non-probability sampling technique, which means that the sample may not be representative of the population. This technique may introduce bias into the study since the participants are self-selected and may not be representative of the entire college student population. Therefore, the results of this study may not be generalizable to other populations.
• Participant Characteristics: The study does not provide any information about the demographic characteristics of the participants, such as age, gender, race, or socioeconomic status. This information is important because social media use and mental health may vary among different demographic groups.
• Data Collection Method: The study used a self-administered survey to collect data. Self-administered surveys may be subject to response bias and may not accurately reflect participants’ actual behaviors and experiences.
• Data Analysis: The study used descriptive statistics and regression analysis to analyze the data. Descriptive statistics provide a summary of the data, while regression analysis is used to examine the relationship between two or more variables. However, the study did not provide information about the statistical significance of the results or the effect sizes.

Overall, while the study provides some insights into the relationship between social media use and mental health among college students, the use of a convenience sampling technique and the lack of information about participant characteristics limit the generalizability of the findings. In addition, the use of self-administered surveys may introduce bias into the study, and the lack of information about the statistical significance of the results limits the interpretation of the findings.

Note*: Above mentioned example is just a sample for students. Do not copy and paste directly into your assignment. Kindly do your own research for academic purposes.

Applications of Evaluating Research

Here are some of the applications of evaluating research:

• Identifying reliable sources : By evaluating research, researchers, students, and other professionals can identify the most reliable sources of information to use in their work. They can determine the quality of research studies, including the methodology, sample size, data analysis, and conclusions.
• Validating findings: Evaluating research can help to validate findings from previous studies. By examining the methodology and results of a study, researchers can determine if the findings are reliable and if they can be used to inform future research.
• Identifying knowledge gaps: Evaluating research can also help to identify gaps in current knowledge. By examining the existing literature on a topic, researchers can determine areas where more research is needed, and they can design studies to address these gaps.
• Improving research quality : Evaluating research can help to improve the quality of future research. By examining the strengths and weaknesses of previous studies, researchers can design better studies and avoid common pitfalls.
• Informing policy and decision-making : Evaluating research is crucial in informing policy and decision-making in many fields. By examining the evidence base for a particular issue, policymakers can make informed decisions that are supported by the best available evidence.
• Enhancing education : Evaluating research is essential in enhancing education. Educators can use research findings to improve teaching methods, curriculum development, and student outcomes.

Purpose of Evaluating Research

Here are some of the key purposes of evaluating research:

• Determine the reliability and validity of research findings : By evaluating research, researchers can determine the quality of the study design, data collection, and analysis. They can determine whether the findings are reliable, valid, and generalizable to other populations.
• Identify the strengths and weaknesses of research studies: Evaluating research helps to identify the strengths and weaknesses of research studies, including potential biases, confounding factors, and limitations. This information can help researchers to design better studies in the future.
• Inform evidence-based decision-making: Evaluating research is crucial in informing evidence-based decision-making in many fields, including healthcare, education, and public policy. Policymakers, educators, and clinicians rely on research evidence to make informed decisions.
• Identify research gaps : By evaluating research, researchers can identify gaps in the existing literature and design studies to address these gaps. This process can help to advance knowledge and improve the quality of research in a particular field.
• Ensure research ethics and integrity : Evaluating research helps to ensure that research studies are conducted ethically and with integrity. Researchers must adhere to ethical guidelines to protect the welfare and rights of study participants and to maintain the trust of the public.

Characteristics Evaluating Research

Characteristics Evaluating Research are as follows:

• Research question/hypothesis: A good research question or hypothesis should be clear, concise, and well-defined. It should address a significant problem or issue in the field and be grounded in relevant theory or prior research.
• Study design: The research design should be appropriate for answering the research question and be clearly described in the study. The study design should also minimize bias and confounding variables.
• Sampling : The sample should be representative of the population of interest and the sampling method should be appropriate for the research question and study design.
• Data collection : The data collection methods should be reliable and valid, and the data should be accurately recorded and analyzed.
• Results : The results should be presented clearly and accurately, and the statistical analysis should be appropriate for the research question and study design.
• Interpretation of results : The interpretation of the results should be based on the data and not influenced by personal biases or preconceptions.
• Generalizability: The study findings should be generalizable to the population of interest and relevant to other settings or contexts.
• Contribution to the field : The study should make a significant contribution to the field and advance our understanding of the research question or issue.

Advantages of Evaluating Research

Evaluating research has several advantages, including:

• Ensuring accuracy and validity : By evaluating research, we can ensure that the research is accurate, valid, and reliable. This ensures that the findings are trustworthy and can be used to inform decision-making.
• Identifying gaps in knowledge : Evaluating research can help identify gaps in knowledge and areas where further research is needed. This can guide future research and help build a stronger evidence base.
• Promoting critical thinking: Evaluating research requires critical thinking skills, which can be applied in other areas of life. By evaluating research, individuals can develop their critical thinking skills and become more discerning consumers of information.
• Improving the quality of research : Evaluating research can help improve the quality of research by identifying areas where improvements can be made. This can lead to more rigorous research methods and better-quality research.
• Informing decision-making: By evaluating research, we can make informed decisions based on the evidence. This is particularly important in fields such as medicine and public health, where decisions can have significant consequences.
• Advancing the field : Evaluating research can help advance the field by identifying new research questions and areas of inquiry. This can lead to the development of new theories and the refinement of existing ones.

Limitations of Evaluating Research

Limitations of Evaluating Research are as follows:

• Time-consuming: Evaluating research can be time-consuming, particularly if the study is complex or requires specialized knowledge. This can be a barrier for individuals who are not experts in the field or who have limited time.
• Subjectivity : Evaluating research can be subjective, as different individuals may have different interpretations of the same study. This can lead to inconsistencies in the evaluation process and make it difficult to compare studies.
• Limited generalizability: The findings of a study may not be generalizable to other populations or contexts. This limits the usefulness of the study and may make it difficult to apply the findings to other settings.
• Publication bias: Research that does not find significant results may be less likely to be published, which can create a bias in the published literature. This can limit the amount of information available for evaluation.
• Lack of transparency: Some studies may not provide enough detail about their methods or results, making it difficult to evaluate their quality or validity.
• Funding bias : Research funded by particular organizations or industries may be biased towards the interests of the funder. This can influence the study design, methods, and interpretation of results.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Chapter Summary & Overview – Writing Guide...

Research Project – Definition, Writing Guide and...

Research Contribution – Thesis Guide

Future Research – Thesis Guide

Informed Consent in Research – Types, Templates...

Data Verification – Process, Types and Examples

Critically Analyzing Information Sources: Critical Appraisal and Analysis

• Critical Appraisal and Analysis

Initial Appraisal : Reviewing the source

• What are the author's credentials--institutional affiliation (where he or she works), educational background, past writings, or experience? Is the book or article written on a topic in the author's area of expertise? You can use the various Who's Who publications for the U.S. and other countries and for specific subjects and the biographical information located in the publication itself to help determine the author's affiliation and credentials.
• Has your instructor mentioned this author? Have you seen the author's name cited in other sources or bibliographies? Respected authors are cited frequently by other scholars. For this reason, always note those names that appear in many different sources.
• Is the author associated with a reputable institution or organization? What are the basic values or goals of the organization or institution?

B. Date of Publication

• When was the source published? This date is often located on the face of the title page below the name of the publisher. If it is not there, look for the copyright date on the reverse of the title page. On Web pages, the date of the last revision is usually at the bottom of the home page, sometimes every page.
• Is the source current or out-of-date for your topic? Topic areas of continuing and rapid development, such as the sciences, demand more current information. On the other hand, topics in the humanities often require material that was written many years ago. At the other extreme, some news sources on the Web now note the hour and minute that articles are posted on their site.

C. Edition or Revision

Is this a first edition of this publication or not? Further editions indicate a source has been revised and updated to reflect changes in knowledge, include omissions, and harmonize with its intended reader's needs. Also, many printings or editions may indicate that the work has become a standard source in the area and is reliable. If you are using a Web source, do the pages indicate revision dates?

D. Publisher

Note the publisher. If the source is published by a university press, it is likely to be scholarly. Although the fact that the publisher is reputable does not necessarily guarantee quality, it does show that the publisher may have high regard for the source being published.

E. Title of Journal

Is this a scholarly or a popular journal? This distinction is important because it indicates different levels of complexity in conveying ideas. If you need help in determining the type of journal, see Distinguishing Scholarly from Non-Scholarly Periodicals . Or you may wish to check your journal title in the latest edition of Katz's Magazines for Libraries (Olin Reference Z 6941 .K21, shelved at the reference desk) for a brief evaluative description.

Critical Analysis of the Content

Having made an initial appraisal, you should now examine the body of the source. Read the preface to determine the author's intentions for the book. Scan the table of contents and the index to get a broad overview of the material it covers. Note whether bibliographies are included. Read the chapters that specifically address your topic. Reading the article abstract and scanning the table of contents of a journal or magazine issue is also useful. As with books, the presence and quality of a bibliography at the end of the article may reflect the care with which the authors have prepared their work.

A. Intended Audience

What type of audience is the author addressing? Is the publication aimed at a specialized or a general audience? Is this source too elementary, too technical, too advanced, or just right for your needs?

B. Objective Reasoning

• Is the information covered fact, opinion, or propaganda? It is not always easy to separate fact from opinion. Facts can usually be verified; opinions, though they may be based on factual information, evolve from the interpretation of facts. Skilled writers can make you think their interpretations are facts.
• Does the information appear to be valid and well-researched, or is it questionable and unsupported by evidence? Assumptions should be reasonable. Note errors or omissions.
• Are the ideas and arguments advanced more or less in line with other works you have read on the same topic? The more radically an author departs from the views of others in the same field, the more carefully and critically you should scrutinize his or her ideas.
• Is the author's point of view objective and impartial? Is the language free of emotion-arousing words and bias?

C. Coverage

• Does the work update other sources, substantiate other materials you have read, or add new information? Does it extensively or marginally cover your topic? You should explore enough sources to obtain a variety of viewpoints.
• Is the material primary or secondary in nature? Primary sources are the raw material of the research process. Secondary sources are based on primary sources. For example, if you were researching Konrad Adenauer's role in rebuilding West Germany after World War II, Adenauer's own writings would be one of many primary sources available on this topic. Others might include relevant government documents and contemporary German newspaper articles. Scholars use this primary material to help generate historical interpretations--a secondary source. Books, encyclopedia articles, and scholarly journal articles about Adenauer's role are considered secondary sources. In the sciences, journal articles and conference proceedings written by experimenters reporting the results of their research are primary documents. Choose both primary and secondary sources when you have the opportunity.

D. Writing Style

Is the publication organized logically? Are the main points clearly presented? Do you find the text easy to read, or is it stilted or choppy? Is the author's argument repetitive?

E. Evaluative Reviews

• Locate critical reviews of books in a reviewing source , such as the Articles & Full Text , Book Review Index , Book Review Digest, and ProQuest Research Library . Is the review positive? Is the book under review considered a valuable contribution to the field? Does the reviewer mention other books that might be better? If so, locate these sources for more information on your topic.
• Do the various reviewers agree on the value or attributes of the book or has it aroused controversy among the critics?
• For Web sites, consider consulting this evaluation source from UC Berkeley .

Permissions Information

If you wish to use or adapt any or all of the content of this Guide go to Cornell Library's Research Guides Use Conditions to review our use permissions and our Creative Commons license.

• Next: Tips >>
• Last Updated: Jun 21, 2024 3:08 PM
• URL: https://guides.library.cornell.edu/critically_analyzing

How to read a paper, critical review

Reading a scientific article is a complex task. The worst way to approach this task is to treat it like the reading of a textbook—reading from title to literature cited, digesting every word along the way without any reflection or criticism.

A critical review (sometimes called a critique, critical commentary, critical appraisal, critical analysis) is a detailed commentary on and critical evaluation of a text. You might carry out a critical review as a stand-alone exercise, or as part of your research and preparation for writing a literature review. The following guidelines are designed to help you critically evaluate a research article.

How to Read a Scientific Article

You should begin by skimming the article to identify its structure and features. As you read, look for the author’s main points.

• Generate questions before, during, and after reading.
• Draw inferences based on your own experiences and knowledge.
• To really improve understanding and recall, take notes as you read.

What is meant by critical and evaluation?

• To be critical does not mean to criticise in an exclusively negative manner. To be critical of a text means you question the information and opinions in the text, in an attempt to evaluate or judge its worth overall.
• An evaluation is an assessment of the strengths and weaknesses of a text. This should relate to specific criteria, in the case of a research article. You have to understand the purpose of each section, and be aware of the type of information and evidence that are needed to make it convincing, before you can judge its overall value to the research article as a whole.

Useful Downloads

• How to read a scientific paper
• How to conduct a critical review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Working with sources
• Evaluating Sources | Methods & Examples

Evaluating Sources | Methods & Examples

Published on June 2, 2022 by Eoghan Ryan . Revised on May 31, 2023.

The sources you use are an important component of your research. It’s important to evaluate the sources you’re considering using, in order to:

• Ensure that they’re credible
• Determine whether they’re relevant to your topic
• Assess the quality of their arguments

Table of contents

Evaluating a source’s credibility, evaluating a source’s relevance, evaluating a source’s arguments, other interesting articles, frequently asked questions about evaluating sources.

Evaluating the credibility of a source is an important way of sifting out misinformation and determining whether you should use it in your research. Useful approaches include the CRAAP test and lateral reading .

One of the best ways to evaluate source credibility is the CRAAP test . This stands for:

• Currency: Does the source reflect recent research?
• Relevance: Is the source related to your research topic?
• Authority: Is it a respected publication? Is the author an expert in their field?
• Accuracy: Does the source support its arguments and conclusions with evidence?
• Purpose: What is the author’s intention?

How you evaluate a source using these criteria will depend on your subject and focus. It’s important to understand the types of sources and how you should use them in your field of research.

Lateral reading

Lateral reading is the act of evaluating the credibility of a source by comparing it to other sources. This allows you to:

• Verify evidence
• Contextualize information
• Find potential weaknesses

If a source is using methods or drawing conclusions that are incompatible with other research in its field, it may not be reliable.

Rather than taking these figures at face value, you decide to determine the accuracy of the source’s claims by cross-checking them with official statistics such as census reports and figures compiled by the Department of Homeland Security’s Office of Immigration Statistics.

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

How you evaluate the relevance of a source will depend on your topic, and on where you are in the research process . Preliminary evaluation helps you to pick out relevant sources in your search, while in-depth evaluation allows you to understand how they’re related.

Preliminary evaluation

As you cannot possibly read every source related to your topic, you can use preliminary evaluation to determine which sources might be relevant. This is especially important when you’re surveying a large number of sources (e.g., in a literature review or systematic review ).

One way to do this is to look at paratextual material, or the parts of a work other than the text itself.

• Look at the table of contents to determine the scope of the work.
• Consult the index for key terms or the names of important scholars.

You can also read abstracts , prefaces , introductions , and conclusions . These will give you a clear idea of the author’s intentions, the parameters of the research, and even the conclusions they draw.

Preliminary evaluation is useful as it allows you to:

• Determine whether a source is worth examining in more depth
• Quickly move on to more relevant sources
• Increase the quality of the information you consume

While this preliminary evaluation is an important step in the research process, you should engage with sources more deeply in order to adequately understand them.

In-depth evaluation

Begin your in-depth evaluation with any landmark studies in your field of research, or with sources that you’re sure are related to your research topic.

As you read, try to understand the connections between the sources. Look for:

• Key debates: What topics or questions are currently influencing research? How does the source respond to these key debates?
• Major publications or critics: Are there any specific texts or scholars that have greatly influenced the field? How does the source engage with them?
• Trends: Is the field currently dominated by particular theories or research methods ? How does the source respond to these?
• Gaps: Are there any oversights or weaknesses in the research?

Even sources whose conclusions you disagree with can be relevant, as they can strengthen your argument by offering alternative perspectives.

Every source should contribute to the debate about its topic by taking a clear position. This position and the conclusions the author comes to should be supported by evidence from direct observation or from other sources.

Most sources will use a mix of primary and secondary sources to form an argument . It is important to consider how the author uses these sources. A good argument should be based on analysis and critique, and there should be a logical relationship between evidence and conclusions.

To assess an argument’s strengths and weaknesses, ask:

• Does the evidence support the claim?
• How does the author use evidence? What theories, methods, or models do they use?
• Could the evidence be used to draw other conclusions? Can it be interpreted differently?
• How does the author situate their argument in the field? Do they agree or disagree with other scholars? Do they confirm or challenge established knowledge?

Situating a source in relation to other sources ( lateral reading ) can help you determine whether the author’s arguments and conclusions are reliable and how you will respond to them in your own writing.

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

• ChatGPT vs human editor
• ChatGPT citations
• Is ChatGPT trustworthy?
• Using ChatGPT for your studies
• What is ChatGPT?
• Chicago style
• Paraphrasing

 Plagiarism

• Types of plagiarism
• Self-plagiarism
• Avoiding plagiarism
• Academic integrity
• Consequences of plagiarism
• Common knowledge

Prevent plagiarism. Run a free check.

As you cannot possibly read every source related to your topic, it’s important to evaluate sources to assess their relevance. Use preliminary evaluation to determine whether a source is worth examining in more depth.

This involves:

• Reading abstracts , prefaces, introductions , and conclusions
• Looking at the table of contents to determine the scope of the work
• Consulting the index for key terms or the names of important scholars

Lateral reading is the act of evaluating the credibility of a source by comparing it with other sources. This allows you to:

A credible source should pass the CRAAP test  and follow these guidelines:

• The information should be up to date and current.
• The author and publication should be a trusted authority on the subject you are researching.
• The sources the author cited should be easy to find, clear, and unbiased.
• For a web source, the URL and layout should signify that it is trustworthy.

The CRAAP test is an acronym to help you evaluate the credibility of a source you are considering using. It is an important component of information literacy .

The CRAAP test has five main components:

• Currency: Is the source up to date?
• Relevance: Is the source relevant to your research?
• Authority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
• Accuracy: Is the source supported by evidence? Are the claims cited correctly?
• Purpose: What was the motive behind publishing this source?

Scholarly sources are written by experts in their field and are typically subjected to peer review . They are intended for a scholarly audience, include a full bibliography, and use scholarly or technical language. For these reasons, they are typically considered credible sources .

Popular sources like magazines and news articles are typically written by journalists. These types of sources usually don’t include a bibliography and are written for a popular, rather than academic, audience. They are not always reliable and may be written from a biased or uninformed perspective, but they can still be cited in some contexts.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Ryan, E. (2023, May 31). Evaluating Sources | Methods & Examples. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/working-with-sources/evaluating-sources/

Is this article helpful?

Eoghan Ryan

Other students also liked, student guide: information literacy | meaning & examples, types of sources explained | examples & tips, what are credible sources & how to spot them | examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Writing Rules
• Running Head & Page numbers
• Using Quotations
• Citing Sources
• Reference List
• General Reference List Principles
• Structure of the Report

Introduction

• References & Appendices
• Unpacking the Assignment Topic
• Planning and Structuring the Assignment
• Writing the Assignment
• Writing Concisely
• Developing Arguments

Critically Evaluating Research

• Editing the Assignment
• Writing in the Third Person
• Directive Words
• Before You Submit
• Cover Sheet & Title Page
• Academic Integrity
• Marking Criteria
• Word Limit Rules
• Submitting Your Work
• Writing Effective E-mails
• Writing Concisely Exercise
• About Redbook

Some research reports or assessments will require you critically evaluate a journal article or piece of research. Below is a guide with examples of how to critically evaluate research and how to communicate your ideas in writing.

To develop the skill of being able to critically evaluate, when reading research articles in psychology read with an open mind and be active when reading. Ask questions as you go and see if the answers are provided. Initially skim through the article to gain an overview of the problem, the design, methods, and conclusions. Then read for details and consider the questions provided below for each section of a journal article.

• Did the title describe the study?
• Did the key words of the title serve as key elements of the article?
• Was the title concise, i.e., free of distracting or extraneous phrases?
• Was the abstract concise and to the point?
• Did the abstract summarise the study’s purpose/research problem, the independent and dependent variables under study, methods, main findings, and conclusions?
• Did the abstract provide you with sufficient information to determine what the study is about and whether you would be interested in reading the entire article?
• Was the research problem clearly identified?
• Is the problem significant enough to warrant the study that was conducted?
• Did the authors present an appropriate theoretical rationale for the study?
• Is the literature review informative and comprehensive or are there gaps?
• Are the variables adequately explained and operationalised?
• Are hypotheses and research questions clearly stated? Are they directional? Do the author’s hypotheses and/or research questions seem logical in light of the conceptual framework and research problem?
• Overall, does the literature review lead logically into the Method section?
• Is the sample clearly described, in terms of size, relevant characteristics (gender, age, SES, etc), selection and assignment procedures, and whether any inducements were used to solicit subjects (payment, subject credit, free therapy, etc)?
• What population do the subjects represent (external validity)?
• Are there sufficient subjects to produce adequate power (statistical validity)?
• Have the variables and measurement techniques been clearly operationalised?
• Do the measures/instruments seem appropriate as measures of the variables under study (construct validity)?
• Have the authors included sufficient information about the psychometric properties (eg. reliability and validity) of the instruments?
• Are the materials used in conducting the study or in collecting data clearly described?
• Are the study’s scientific procedures thoroughly described in chronological order?
• Is the design of the study identified (or made evident)?
• Do the design and procedures seem appropriate in light of the research problem, conceptual framework, and research questions/hypotheses?
• Are there other factors that might explain the differences between groups (internal validity)?
• Were subjects randomly assigned to groups so there was no systematic bias in favour of one group? Was there a differential drop-out rate from groups so that bias was introduced (internal validity and attrition)?
• Were all the necessary control groups used? Were participants in each group treated identically except for the administration of the independent variable?
• Were steps taken to prevent subject bias and/or experimenter bias, eg, blind or double blind procedures?
• Were steps taken to control for other possible confounds such as regression to the mean, history effects, order effects, etc (internal validity)?
• Were ethical considerations adhered to, eg, debriefing, anonymity, informed consent, voluntary participation?
• Overall, does the method section provide sufficient information to replicate the study?
• Are the findings complete, clearly presented, comprehensible, and well organised?
• Are data coding and analysis appropriate in light of the study’s design and hypotheses? Are the statistics reported correctly and fully, eg. are degrees of freedom and p values given?
• Have the assumptions of the statistical analyses been met, eg. does one group have very different variance to the others?
• Are salient results connected directly to hypotheses? Are there superfluous results presented that are not relevant to the hypotheses or research question?
• Are tables and figures clearly labelled? Well-organised? Necessary (non-duplicative of text)?
• If a significant result is obtained, consider effect size. Is the finding meaningful? If a non-significant result is found, could low power be an issue? Were there sufficient levels of the IV?
• If necessary have appropriate post-hoc analyses been performed? Were any transformations performed; if so, were there valid reasons? Were data collapsed over any IVs; if so, were there valid reasons? If any data was eliminated, were valid reasons given?

Discussion and Conclusion

• Are findings adequately interpreted and discussed in terms of the stated research problem, conceptual framework, and hypotheses?
• Is the interpretation adequate? i.e., does it go too far given what was actually done or not far enough? Are non-significant findings interpreted inappropriately?
• Is the discussion biased? Are the limitations of the study delineated?
• Are implications for future research and/or practical application identified?
• Are the overall conclusions warranted by the data and any limitations in the study? Are the conclusions restricted to the population under study or are they generalised too widely?
• Is the reference list sufficiently specific to the topic under investigation and current?
• Are citations used appropriately in the text?

General Evaluation

• Is the article objective, well written and organised?
• Does the information provided allow you to replicate the study in all its details?
• Was the study worth doing? Does the study provide an answer to a practical or important problem? Does it have theoretical importance? Does it represent a methodological or technical advance? Does it demonstrate a previously undocumented phenomenon? Does it explore the conditions under which a phenomenon occurs?

How to turn your critical evaluation into writing

Example from a journal article.

ON YOUR 1ST ORDER

How To Write A Critical Analysis Of A Research Paper

By Laura Brown on 29th May 2023

Conducting a critical analysis of a research paper includes the evaluation of its methodology, data sources, and findings. Alongside, it is necessary to assess the paper’s strengths and weaknesses, identify any biases or limitations, and examine its contribution to the respective field. Additionally, considering alternative interpretations and potential implications is key to providing a comprehensive analysis.

The art of critical analysis is a crucial skill for researchers and scholars alike. It allows us to delve deeper, question assumptions, and uncover the strengths and weaknesses of a research paper. This blog covers the essential steps to master the art of conducting a critical evaluation along with the examples.

Research papers serve as a foundation for advancing knowledge and shaping academic discourse. By critically analysing these papers, we can assess their validity, identify their contributions, and even influence the direction of future research. Throughout this post, we will guide you through the process of understanding a research paper, evaluating its strengths and weaknesses, assessing its contribution, formulating your analysis, considering alternative perspectives, and providing recommendations.

Whether you’re a student, a researcher, or an avid reader of scholarly work, developing the ability to critically analyse a research paper will enhance your understanding and engagement with academic literature and scientific articles. Let’s dive into the world of critical analysis and unlock the secret insights as you buy research paper from us or read this handy guide.

1. Understand The Research Paper

To effectively analyse a research paper, it is crucial to gain a comprehensive understanding of its content. You may begin by thoroughly reading the paper and paying attention to every detail. Further, you should identify the main research question or objective that the study aims to address. This will provide you with a focal point for your analysis.

Now, familiarise yourself with the methodology used and the data collected for the research. Moreover, evaluate the appropriateness and reliability of the chosen methodology, and assess the quality of the data collection and analysis. Understanding these aspects will help you gauge the validity and firmness of the research.

Additionally, take note of the key findings and conclusions presented in the paper and Analyse the supporting evidence along with evaluating the conclusions align with the research objectives. You should also consider any limitations or potential biases that might affect the interpretation of the results. By thoroughly understanding the scientific paper, you will lay a solid foundation for your critical analysis. In case you face any difficulty understanding the paper, you can always contact research paper service anytime, we will definitely help you.

2. Identify The Strengths And Weaknesses

In order to conduct a comprehensive critical analysis on research paper, it is essential to identify its strengths and weaknesses . Here are key aspects to consider during this evaluation process.

a. Evaluate The Research Paper’s Structure

First, assess whether it follows a logical flow and if the sections are well-developed and interconnected. Remember, a well-structured paper enhances readability and comprehension.

b. Assess The Clarity Of The Arguments

Next, look for concise statements and a logical progression of ideas. Moreover, analyse how well the author supports their arguments with relevant evidence and whether the reasoning is sound.

c. Analyse The Authenticity Of The Facts & Figures

Further, analyse the relevance of the data and sources used. You should examine the quality and appropriateness of the cited sources . Also, look at the facts presented if they adequately supports the claims made by the author and whether there is a robust foundation for the conclusions drawn.

d. Identifying Potential Limitations

Now, this is the time to identify the strengths and weaknesses of the research methods used. At this moment, you should also consider any limitations that may impact the validity or generalizability of the findings.

e. Determine Biases Or Conflicts Of Interest

Finally, consider the author’s affiliations, funding sources, or personal beliefs that could influence the research outcomes.

3. Evaluate The Research Paper’s Contribution

It is crucial to have a deeper look into the contribution while critically analysing a research paper. You may go through the following steps for critical evaluation.

a. Assess The Significance Of The Research

Firstly, determine whether the paper presents new ideas, approaches, or insights that contribute to the field. Additonally, assess its potential to advance knowledge and fill gaps in existing research.

b. Consider The Research Paper’s Contribution

Secondly, evaluate how the paper builds upon or challenges existing theories, concepts, or methodologies along with assessing its potential to expand understanding or provide novelty.

c. Find Potential Impact Of The Findings

Finally, analyse how the research paper’s findings may influence practice, policy, or future research directions. Also, consider the broader implications and relevance of the research within the context of the field or society.

4. Formulate Your Analysis

Formulating a strong and insightful analysis is a crucial aspect of research paper critical analysis. To effectively present your analysis, follow the below-mentioned steps:

• Begin by developing a clear thesis statement that reflects your overall analysis of the research paper. This statement should encapsulate the main point or argument you wish to convey.
• Next, support your analysis with specific examples from the research paper. Referencing specific sections, findings, or arguments helps substantiate your points and provides evidence for your analysis.
• Ensure that you present your analysis in a logical and organised manner. Structure your analysis in a way that flows coherently, with each point building upon the previous one. To achieve this, use clear and concise language that conveys your thoughts effectively.

Let’s see a critical analysis research paper example for initiating your analysis with a thesis statement.

The research paper’s findings on the impact of deforestation are valuable, but its failure to address socio-economic factors limits its comprehensive understanding of the issue.

5. Consider Alternative Perspectives

In a critical analysis of a scientific article or research paper it is essential to consider alternative perspectives to present a well-rounded evaluation. Follow these steps to effectively engage with different viewpoints.

• Start by acknowledging and discussing alternative interpretations or viewpoints that exist regarding the research paper. This demonstrates your openness to diverse perspectives and fosters a comprehensive analysis.
• Next, compare and contrast these different perspectives with your own analysis. Identify areas of agreement or disagreement and highlight any significant differences in the interpretation of the research findings or methodology.
• Provide reasoning and evidence to support your stance in the critical analysis. Present logical arguments and use relevant evidence to justify your perspective. Consider the strengths and weaknesses of alternative viewpoints and explain why you find your analysis to be more compelling.

Certainly! Here’s a critical evaluation of a research paper example for considering alternative perspectives in the context of a research paper on climate change:

It becomes evident that the paper’s findings on the impact of deforestation are valuable. The research provides insights into the ecological consequences and loss of biodiversity resulting from deforestation. However, a crucial limitation of the paper lies in its failure to address socio-economic factors. By neglecting the socio-economic aspects, such as the role of industries, government policies, and societal behaviours, the research paper lacks a comprehensive understanding of the issue. To gain a holistic understanding, it is recommended to consult the following additional resources.

Here you can present various resources as you need.

6. Provide Recommendations Or Suggestions

Considering critical analysis in a research paper, it is important to go beyond evaluating the strengths and weaknesses and offer constructive recommendations for improvement. Here’s a research paper example of how this section could be written.

Based on the critical analysis of the research paper on renewable energy sources, several recommendations emerge. Firstly, the paper could benefit from a more comprehensive discussion of the economic viability of renewable technologies. Incorporating an analysis of cost-effectiveness and potential financing models would strengthen the paper’s practical implications. Secondly, the authors should consider addressing potential limitations and uncertainties associated with the data sources used. Providing transparency and acknowledging any gaps would enhance the overall credibility of the research. Lastly, there is a need for further investigation into the social acceptance and adoption of renewable energy technologies, as understanding the human dimension is crucial for successful implementation. By offering these recommendations, the research paper can be enhanced and contribute more effectively to the field.

7. Writing The Conclusion

Students often ask how to write the conclusion of a report and critical analysis; here is how it is done. The conclusion of a critical analysis of scientific literature or research paper should succinctly summarise the key points and analysis, emphasising the significance of critical thinking. It should reinforce the importance of addressing any limitations or gaps in the research and encourage further exploration. The conclusion should leave readers with a clear understanding of the paper’s strengths and weaknesses, and inspire them to apply critical analysis principles in their own research endeavours. Here is an example of critical analysis of a research paper in regards to conclusion.

The critical analysis of the research paper on climate change brings to light the importance of addressing socio-economic factors for a comprehensive understanding of the issue. While the paper’s findings on the impact of deforestation are valuable, the omission of socio-economic considerations limits its applicability in developing effective solutions. It is crucial for future research to incorporate the interplay between environmental and socio-economic factors to devise holistic strategies. By recognising and rectifying these gaps, researchers can contribute to a more nuanced understanding of climate change and inform policies that foster sustainable development and resilience.

8. Additional Resources Or References

For readers seeking further exploration and a deeper understanding of the research paper, you can also put up some additional resources . However, this is not the part of the critical analysis, but still you can include it.

Summing Up The Tips For You

Here are 10 points for you as a summary of this blog. You may also consider it as a critical analysis of a research paper checklist while you prepare to conduct it.

• Thoroughly read the research paper to gain a deep understanding of its content.
• Evaluate the research question/objective and assess its relevance and significance.
• Assess the methodology and data used, considering their validity and reliability.
• Analyse the clarity and coherence of the arguments presented in the paper.
• Give a keen look at the relevance and sufficiency of the evidence and sources used.
• Critique the limitations and potential biases of the research.
• Consider alternative perspectives and compare them with your analysis.
• Assess the originality and contribution of the research to the existing knowledge.
• Examine the implications and potential impact of the research findings.
• Provide constructive criticism and suggestions for improvement, including areas for further research or investigation.

Follow this research paper checklist for critically analysing a research paper, and you will definitely rock it.

Laura Brown, a senior content writer who writes actionable blogs at Crowd Writer.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Towards reliable healthcare llm agents: a case study for pilgrims during hajj.

1. Introduction

• Knowledge retrieval: When the model encounters uncertain or ambiguous input, the RAG module retrieves relevant knowledge from specific external resources. This retrieval process enables the model to augment its understanding of the topic at hand and generate more informed responses.
• Validation of uncertain text: After retrieving relevant knowledge, the RAG module validates the uncertain text generated by the GPT-3.5 Turbo model against the retrieved information. By cross-referencing the model’s output with external knowledge sources, the RAG module assesses the accuracy and credibility of the generated text, identifying and correcting any inaccuracies or inconsistencies before finalizing the response.
• Domain-specific fine-tuning of LLM: We fine-tune a large language model (LLM) specifically for the domain of healthcare and cultural sensitivities relevant to Hajj pilgrims. This fine-tuning process ensures that the model is capable of understanding and generating relevant responses within the context of healthcare conversations during the pilgrimage.
• Introducing the HajjHealthQA dataset: To facilitate the development and evaluation of our healthcare chatbot, we introduce the HajjHealthQA dataset. This dataset contains a diverse collection of questions, answers, and conversations relevant to healthcare issues faced by Hajj pilgrims. We also employ synthetic data augmentation techniques ( https://github.com/AbeerMostafa/HajjHealthQA-Dataset (accessed on 1 March 2024)).
• RAG module for uncertainty validation: We add a retrieval-augmented generation (RAG) module to validate uncertain information provided by the chatbot. This mechanism enhances the reliability and accuracy of the chatbot’s responses by cross-referencing generated text with external knowledge sources.
• Training a secondary AI agent on the HealthVer dataset: We train two separate models as part of our framework, one on the HajjHealthQA dataset for Hajj-specific healthcare inquiries and another on the HealthVer dataset for medical information verification. The latter is used to verify that the medical information generated by our chatbot is supported by medical evidence.
• Prompt engineering for case study specifics: We employ prompt engineering techniques tailored to the specific case study of building a healthcare chatbot for Hajj pilgrims. This ensures that the chatbot’s responses are optimized for relevance, accuracy, and cultural appropriateness within the context of Hajj-related healthcare scenarios.
• Multilingual support: To accommodate the linguistic diversity of Hajj pilgrims, our chatbot offers multilingual support, allowing users to interact in their preferred language.

2. Related Work

2.1. health challenges faced by hajj pilgrims, 2.2. medical q&a, 2.3. use of synthetic data, 2.4. hajj q&a, 3. hajjhealthqa dataset, 4. methodology, 4.1. model fine-tuning, 4.2. retrieval-augmented generation, 4.3. evidence-based verification, 4.4. prompt engineering.

• Task-specific prompts
• Multilingual support
• Customization for cultural sensitivity
• Contextual awareness and follow-up prompts
• Iterative improvement through user feedback

5. Experimental Setup

5.1. hyperparameter tuning, 5.1.1. number of epochs, 5.1.2. batch size, 5.1.3. learning rate multiplier, 5.2. evaluation metrics, 6. results and discussion, 6.1. accuracy analysis, 6.2. quality metrics, 6.3. comparison with benchmark results, 7. data privacy and ethical considerations, 8. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Abdelmoety, D.; El-Bakri, N.; Almowalld, W.; Turkistani, Z.; Bugis, B.; Baseif, E.; Melbari, M.H.; AlHarbi, K.; Abu-Shaheen, A. Characteristics of Heat Illness during Hajj: A Cross-Sectional Study. BioMed Res. Int. 2018 , 2018 , 5629474. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Al-Masud, S.M.R.; Bakar, A.A.; Yussof, S. Determining the types of diseases and emergency issues in Pilgrims during Hajj: A literature review. Int. J. Adv. Comput. Sci. Appl. 2016 , 7 , 87–94. [ Google Scholar ]
• Razavi, S.M.; Mardani, M.; Salamati, P. Infectious diseases and preventive measures during hajj mass gatherings: A review of the literature. Arch. Clin. Infect. Dis. 2018 , 13 , e62526. [ Google Scholar ] [ CrossRef ]
• Salmon-Rousseau, A.; Piednoir, E.; Cattoir, V.; de La Blanchardiere, A. Hajj-associated infections. MEdecine Mal. Infect. 2016 , 46 , 346–354. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Yezli, S.; Yassin, Y.; Mushi, A.; Almuzaini, Y.; Khan, A. Pattern of utilization, disease presentation, and medication prescribing and dispensing at 51 primary healthcare centers during the Hajj mass gathering. BMC Health Serv. Res. 2022 , 22 , 143. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Shen, Y.; Heacock, L.; Elias, J.; Hentel, K.D.; Reig, B.; Shih, G.; Moy, L. ChatGPT and other large language models are double-edged swords. Radiology 2023 , 307 , e230163. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Javaid, M.; Haleem, A.; Singh, R.P. ChatGPT for healthcare services: An emerging stage for an innovative perspective. BenchCounc. Trans. Benchmarks Stand. Eval. 2023 , 3 , 100105. [ Google Scholar ] [ CrossRef ]
• De Angelis, L.; Baglivo, F.; Arzilli, G.; Privitera, G.P.; Ferragina, P.; Tozzi, A.E.; Rizzo, C. ChatGPT and the rise of large language models: The new AI-driven infodemic threat in public health. Front. Public Health 2023 , 11 , 1166120. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; Volume 30. [ Google Scholar ]
• Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I. Improving Language Understanding by Generative Pre-Training. 2018. Available online: https://www.mikecaptain.com/resources/pdf/GPT-1.pdf (accessed on 5 February 2024).
• Glik, D.C. Risk communication for public health emergencies. Annu. Rev. Public Health 2007 , 28 , 33–54. [ Google Scholar ] [ CrossRef ]
• Almehmadi, M.; Pescaroli, G.; Alqahtani, J.S.; Oyelade, T. Investigating health risk perceptions during the Hajj: Pre-Travel advice and adherence to preven-tative health measures. Afr. J. Respir. Med. 2021 , 16 , 1–6. [ Google Scholar ]
• Alqahtani, A.S.; Tashani, M.; Heywood, A.E.; Booy, R.; Rashid, H.; Wiley, K.E. Exploring Australian Hajj Tour Operators’ Knowledge and Practices Regarding Pilgrims’ Health Risks: A Qualitative Study. JMIR Public Health Surveill. 2019 , 5 , e10960. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Aljohani, A.; Nejaim, S.; Khayyat, M.; Aboulola, O. E-government and logistical health services during Hajj season. Bull. Natl. Res. Cent. 2022 , 46 , 112. [ Google Scholar ] [ CrossRef ]
• Dzaraly, N.D.; Rahman, N.I.A.; Simbak, N.B.; Ab Wahab, S.; Osman, O.; Ismail, S.; Haque, M. Patterns of communicable and non-communicable diseases in Pilgrims during Hajj. Res. J. Pharm. Technol. 2014 , 7 , 12. [ Google Scholar ]
• Abdelhay, M.; Mohammed, A.; Hefny, H.A. Deep learning for Arabic healthcare: MedicalBot. Soc. Netw. Anal. Min. 2023 , 13 , 71. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Singh, S.; Susan, S. Healthcare Question–Answering System: Trends and Perspectives. In Proceedings of the International Health Informatics Conference: IHIC 2022, Cuttack, India, 17–19 May 2022; Springer: Cham, Switzerland, 2023; pp. 239–249. [ Google Scholar ]
• Pal, V.K.; Singh, S.; Sinha, A.; Shekh, M.S. Medical Chatbot using AI and NLP. i-Manag. J. Softw. Eng. 2022 , 16 , 46. [ Google Scholar ]
• Long, C.; Subburam, D.; Lowe, K.; Santos, A.d.; Zhang, J.; Hwang, S.; Saduka, N.; Horev, Y.; Su, T.; Cote, D.; et al. ChatENT: Augmented Large Language Model for Expert Knowledge Retrieval in Otolaryngology-Head and Neck Surgery. medRxiv 2023 , 2023-08. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chen, S.; Li, Y.; Lu, S.; Van, H.; Aerts, H.J.W.L.; Savova, G.K.; Bitterman, D.S. Evaluating the ChatGPT family of models for biomedical reasoning and classification. J. Am. Med. Inform. Assoc. JAMIA 2024 , 31 , 940–948. [ Google Scholar ] [ CrossRef ]
• Puri, R.; Spring, R.; Shoeybi, M.; Patwary, M.; Catanzaro, B. Training Question Answering Models From Synthetic Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16–20 November 2020; Webber, B., Cohn, T., He, Y., Liu, Y., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 5811–5826. [ Google Scholar ] [ CrossRef ]
• Wei, J.; Huang, D.; Lu, Y.; Zhou, D.; Le, Q.V. Simple synthetic data reduces sycophancy in large language models. arXiv 2023 , arXiv:2308.03958. [ Google Scholar ]
• Sulaiman, S.; Mohamed, H.; Arshad, M.R.M.; Rashid, N.A.A.; Yusof, U.K. Hajj-QAES: A Knowledge-Based Expert System to Support Hajj Pilgrims in Decision Making. In Proceedings of the 2009 International Conference on Computer Technology and Development, Kota Kinabalu, Malaysia, 13–15 November 2009; Volume 1, pp. 442–446. [ Google Scholar ] [ CrossRef ]
• Sharef, N.M.; Murad, M.A.; Mustapha, A.; Shishechi, S. Semantic question answering of umrah pilgrims to enable self-guided education. In Proceedings of the 2013 13th International Conference on Intellient Systems Design and Applications, Salangor, Malaysia, 8–10 December 2013; pp. 141–146. [ Google Scholar ] [ CrossRef ]
• Mohamed, H.H.; Arshad, M.R.H.M.; Azmi, M.D. M-HAJJ DSS: A mobile decision support system for Hajj pilgrims. In Proceedings of the 2016 3rd International Conference on Computer and Information Sciences (ICCOINS), Kuala Lumpur, Malaysia, 15–17 August 2016; pp. 132–136. [ Google Scholar ] [ CrossRef ]
• Nusuk: Your Official Guide to Makkah and Madinah. Available online: https://www.nusuk.sa/ (accessed on 1 November 2023).
• Mecca WABot: Smart System Makes Hajj and Umrah Pilgrims Easy to Worship. Available online: https://kumparan.com/beritaanaksurabaya/mecca-wabot-sistem-pintar-mudahkan-jemaah-haji-dan-umrah-beribadah-20f6faH8EMZ/2 (accessed on 1 November 2023).
• Ministry of Health in the Kingdom of Saudi Arabia. Available online: https://www.moh.gov.sa/en/ (accessed on 1 November 2023).
• WHO Chronic Respiratory Diseases. Available online: https://www.who.int/health-topics/chronic-respiratory-diseases#tab=tab_3 (accessed on 1 November 2023).
• Ministry of Hajj and Umrah in the Kingdom of Saudi Arabia. Available online: https://www.haj.gov.sa/Home (accessed on 1 November 2023).
• CGD Society—FAQ Lung Issues. Available online: https://cgdsociety.org/living-with-cgd/managing-cgd/common-problems/lung-problems/faqs-lung-issues/ (accessed on 1 November 2023).
• Top Doctors—Frequently Asked Questions about Lung Diseases. Available online: https://www.topdoctors.co.uk/medical-articles/frequently-asked-questions-about-lung-diseases# (accessed on 1 November 2023).
• Hajj and Umrah Health Requirements. Available online: https://www.saudiembassy.net/hajj-and-umrah-health-requirements. (accessed on 1 November 2023).
• Health Requirements for Hajj. Available online: https://www.moh.gov.sa/en/HealthAwareness/Pilgrims_Health/Pages/default.aspx (accessed on 1 November 2023).
• Sarrouti, M.; Abacha, A.B.; M’rabet, Y.; Demner-Fushman, D. Evidence-based fact-checking of health-related claims. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, 16–20 November 2021; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 3499–3512. [ Google Scholar ]
• Phatak, A.; Mago, V.K.; Agrawal, A.; Inbasekaran, A.; Giabbanelli, P.J. Narrating Causal Graphs with Large Language Models. arXiv 2024 , arXiv:2403.07118. [ Google Scholar ]
• Gao, M.; Hu, X.; Ruan, J.; Pu, X.; Wan, X. LLM-based NLG Evaluation: Current Status and Challenges. arXiv 2024 , arXiv:2402.01383. [ Google Scholar ]
• Saadany, H.; Orǎsan, C. BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-Oriented Text. In Proceedings of the Translation and Interpreting Technology Online Conference, Online, 5–7 July 2021; pp. 48–56. [ Google Scholar ]
• Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.Q.; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 30 April 2020. [ Google Scholar ]
• Akter, S.N.; Yu, Z.; Muhamed, A.; Ou, T.; Bäuerle, A.; Cabrera, Á.A.; Dholakia, K.; Xiong, C.; Neubig, G. An In-depth Look at Gemini’s Language Abilities. arXiv 2023 , arXiv:2312.11444. [ Google Scholar ]

Click here to enlarge figure

Share and Cite

Alghamdi, H.M.; Mostafa, A. Towards Reliable Healthcare LLM Agents: A Case Study for Pilgrims during Hajj. Information 2024 , 15 , 371. https://doi.org/10.3390/info15070371

Alghamdi HM, Mostafa A. Towards Reliable Healthcare LLM Agents: A Case Study for Pilgrims during Hajj. Information . 2024; 15(7):371. https://doi.org/10.3390/info15070371

Alghamdi, Hanan M., and Abeer Mostafa. 2024. "Towards Reliable Healthcare LLM Agents: A Case Study for Pilgrims during Hajj" Information 15, no. 7: 371. https://doi.org/10.3390/info15070371

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems

• Tran, Hung Vinh
• Nguyen, Quoc Viet Hung
• Cui, Lizhen
• Yin, Hongzhi

Since the creation of the Web, recommender systems (RSs) have been an indispensable mechanism in information filtering. State-of-the-art RSs primarily depend on categorical features, which ecoded by embedding vectors, resulting in excessively large embedding tables. To prevent over-parameterized embedding tables from harming scalability, both academia and industry have seen increasing efforts in compressing RS embeddings. However, despite the prosperity of lightweight embedding-based RSs (LERSs), a wide diversity is seen in evaluation protocols, resulting in obstacles when relating LERS performance to real-world usability. Moreover, despite the common goal of lightweight embeddings, LERSs are evaluated with a single choice between the two main recommendation tasks -- collaborative filtering and content-based recommendation. This lack of discussions on cross-task transferability hinders the development of unified, more scalable solutions. Motivated by these issues, this study investigates various LERSs' performance, efficiency, and cross-task transferability via a thorough benchmarking process. Additionally, we propose an efficient embedding compression method using magnitude pruning, which is an easy-to-deploy yet highly competitive baseline that outperforms various complex LERSs. Our study reveals the distinct performance of LERSs across the two tasks, shedding light on their effectiveness and generalizability. To support edge-based recommendations, we tested all LERSs on a Raspberry Pi 4, where the efficiency bottleneck is exposed. Finally, we conclude this paper with critical summaries of LERS performance, model selection suggestions, and underexplored challenges around LERSs for future research. To encourage future research, we publish source codes and artifacts at \href{this link}{https://github.com/chenxing1999/recsys-benchmark}.

• Computer Science - Information Retrieval;
• Computer Science - Machine Learning

• Computer Vision
• Federated Learning
• Reinforcement Learning
• Natural Language Processing
• New Releases
• Advisory Board Members
• 🐝 Partnership and Promotion

To further illustrate the impact of OLMES, the researchers evaluated popular benchmark tasks such as ARC-Challenge, OpenBookQA, and MMLU. The findings showed that models evaluated using OLMES performed better and exhibited reduced discrepancies in reported performance across different references. For instance, the Llama3-70B model achieved a remarkable 93.7% accuracy on the ARC-Challenge task using the multiple-choice format, compared to only 69.0% with the cloze format. This substantial difference underscores the importance of using standardized evaluation practices to obtain reliable results.

In conclusion, the problem of inconsistent evaluations in AI research has been effectively addressed by the introduction of OLMES. The new standard offers a comprehensive solution by standardizing evaluation practices and providing detailed guidelines for all aspects of the evaluation process. Researchers from the Allen Institute for Artificial Intelligence have demonstrated that OLMES improves the reliability of performance measurements and supports meaningful comparisons across different models. By adopting OLMES, the AI community can achieve greater transparency, reproducibility, and fairness in evaluating language models. This advancement is expected to drive further progress in AI research and development, fostering innovation and collaboration among researchers and developers.

Check out the Paper . All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on  Twitter . 

Join our  Telegram Channel and  LinkedIn Gr oup .

If you like our work, you will love our  newsletter..

Don’t Forget to join our  45k+ ML SubReddit

Nikhil is an intern consultant at Marktechpost. He is pursuing an integrated dual degree in Materials at the Indian Institute of Technology, Kharagpur. Nikhil is an AI/ML enthusiast who is always researching applications in fields like biomaterials and biomedical science. With a strong background in Material Science, he is exploring new advancements and creating opportunities to contribute.

Fact or Fiction? NOCHA: A New Benchmark for Evaluating Long-Context Reasoning in LLMs

• EAGLE-2: An Efficient and Lossless Speculative Sampling Method Achieving Speedup Ratios 3.05x - 4.26x which is 20% - 40% Faster than EAGLE-1
• Google Project Zero Introduces Naptime: An Architecture for Evaluating Offensive Security Capabilities of Large Language Models
• LongRAG: A New Artificial Intelligence AI Framework that Combines RAG with Long-Context LLMs to Enhance Performance

RELATED ARTICLES MORE FROM AUTHOR

A deep dive into group relative policy optimization (grpo) method: enhancing mathematical reasoning in open language models, what if we could universally edit any two pieces of dna meet ‘bridge editing’ and ‘bridge rna’: a modular approach to rna-guided genetic rearrangements..., imbue team trains 70b-parameter model from scratch: innovations in pre-training, evaluation, and infrastructure for advanced ai performance, q*: a versatile artificial intelligence ai approach to improve llm performance in reasoning tasks, dolphin{anty} antidetect browser: the ultimate antidetect browser for online anonymity and multi-account management, a deep dive into group relative policy optimization (grpo) method: enhancing mathematical reasoning in..., what if we could universally edit any two pieces of dna meet ‘bridge editing’..., imbue team trains 70b-parameter model from scratch: innovations in pre-training, evaluation, and infrastructure for....

• AI Magazine
• Privacy & TC
• Cookie Policy

🐝 🐝 Join the Fastest Growing AI Research Newsletter...

Thank You 🙌

Privacy Overview

Deep reinforcement learning-based scheduling in distributed systems: a critical review

• Published: 26 June 2024

Cite this article

• Zahra Jalali Khalil Abadi 1 ,
• Najme Mansouri 1 &
• Mohammad Masoud Javidi 1  

Explore all metrics

Many fields of research use parallelized and distributed computing environments, including astronomy, earth science, and bioinformatics. Due to an increase in client requests, service providers face various challenges, such as task scheduling, security, resource management, and virtual machine migration. NP-hard scheduling problems require a long time to implement an optimal or suboptimal solution due to their large solution space. With recent advances in artificial intelligence, deep reinforcement learning (DRL) can be used to solve scheduling problems. The DRL approach combines the strength of deep learning and neural networks with reinforcement learning’s feedback-based learning. This paper provides a comprehensive overview of DRL-based scheduling algorithms in distributed systems by categorizing algorithms and applications. As a result, several articles are assessed based on their main objectives, quality of service and scheduling parameters, as well as evaluation environments (i.e., simulation tools, real-world environment). The literature review indicates that algorithms based on RL, such as Q-learning, are effective for learning scaling and scheduling policies in a cloud environment. Additionally, the challenges and directions for further research on deep reinforcement learning to address scheduling problems were summarized (e.g., edge intelligence, ideal dynamic task scheduling framework, human–machine interaction, resource-hungry artificial intelligence (AI) and sustainability).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Availability of data and materials

Not applicable.

Henderson P, Islam R, Bachman P, Pineau J, Precup D, Meger D (2018) Deep reinforcement learning that matters. In: 32nd AAAI conference on artificial intelligence

Zhao H, Dong C, Cao J, Chen Q (2024) A survey on deep reinforcement learning approaches for traffic signal control. Eng Appl Artif Intell 133:108100

Article   Google Scholar  

Khurana D, Koli A, Khatter K, Singh S (2023) Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl 82:3713–3744

Xiang X, Foo S (2021) Recent advances in deep reinforcement learning applications for solving partially observable Markov Decision Processes (POMDP) problems: Part 1—fundamentals and applications in games, robotics and natural language processing. Machin Learn Knowl Extr 3:554–581

Kaushik P, Sharma AR (2017) Literature survey of statistical, deep and reinforcement learning in natural language processing. In: International conference on computing, communication and automation (ICCCA)

Capra L, Köhler-Bußmeier M (2024) Modular rewritable petri nets: an efficient model for dynamic distributed systems. Theoret Comput Sci 990:114397

Article   MathSciNet   Google Scholar  

Pirani M, Mitra A, Sundaram S (2023) Graph-theoretic approaches for analyzing the resilience of distributed control systems: a tutorial and survey. Automatica 157:111264

Ahmad S, Shakeel I, Mehfuz S, Ahmad J (2023) Deep learning models for cloud, edge, fog, and IoT computing paradigms: survey, recent advances, and future directions. Comput Sci Rev 49:100568

Yin X, Chen G, Qian H, Li L, Su S, Jiang H, Zhang H (2024) A distributed parallel computing-based CFD analysis of plate-type fuel assemblies in CARR reactors. Int Commun Heat Mass Transf 153:107335

Qazi F, Kwak D, Gul Khan F, Ali F, Ullah Khan S (2024) service level agreement in cloud computing: taxonomy, prospects, and challenges. Internet Things 25:101126

Ullah A, Nawi NM, Ouhame S (2022) Recent advancement in VM task allocation system for cloud computing: review from 2015 to 2021. Artif Intell Rev 55:2529–2573

Mohammad Hasani Zade B, Mansouri N (2022) Improved red fox optimizer with fuzzy theory and game theory for task scheduling in cloud environment. J Comput Sci 63:101805

Mohammad Hasani Zade B, Mansouri N, Javidi MM (2022) A two-stage scheduler based on new caledonian crow learning algorithm and reinforcement learning strategy for cloud environment. J Netw Comput Appl 202:103385

Menaka M, Sendhil Kumar KS (2022) Workflow scheduling in cloud environment-challenges, tools, limitations & methodologies: a review. Meas Sens 24:100436

Mohammad Hasani Zade B, Mansouri N, Javidi MM (2021) Multi-objective scheduling technique based on hybrid hitchcock bird algorithm and fuzzy signature in cloud computing. Eng Appl Artif Intell 104:104372

Mohammad Hasani Zade B, Mansouri N, Javidi MM (2021) SAEA: a security-aware and energy-aware task scheduling strategy by parallel squirrel search algorithm in cloud environment. Expert Syst Appl 176:114915

Cao K, Liu Y, Meng G, Sun Q (2020) An overview on edge computing research. IEEE Access 8:85715–85728

Google Scholar  

Satyanarayanan M (2017) The emergence of edge computing. Computer 50:30–39

Shi W, Cao J, Zhang Q, Li Y, Xu L (2016) Edge computing: vision and challenges. IEEE Internet Things J 3:637–646

Jalali Khalil Abadi Z, Mansouri N, Khalouie M (2023) Task scheduling in fog environment—challenges, tools & methodologies: a review. Comput Sci Rev 48:100550

Ul Islam MS, Kumar A, Hu YC (2021) Context-aware scheduling in fog computing: a survey, taxonomy, challenges and future directions. J Netw Comput Appl 180:103008

https://www.educba.com/fog-computing-architecture/

Mahesh B (2020) Machine learning algorithms—a review. Int J Sci Res 9:381–386

Aqib M, Kumar D, Tripathi S (2022) Machine learning for fog computing: Review, opportunities and a fog application classifier and scheduler. Wirel Pers Commun 129:853–880

Kober J, Bagnell JA, Peters J (2013) Reinforcement learning in robotics: a survey. Int J Robot Res 32:1238–1274

Iftikhar S, Gill SS, Song C, Xu M, Aslanpour MS, Toosi AN, Du J, Wu H, Ghosh S, Abdelmoniem AM, Cuadrado F, Varghese B, Rana O, Dustdar S, Uhlig S (2023) AI-based fog and edge computing: a systematic review, taxonomy and future directions. Internet Things 21:100674

Lin B (2024) Reinforcement learning and bandits for speech and language processing: tutorial, review and outlook. Expert Syst Appl 238:122254

Chen X, Zhang H, Wu C, Mao S, Ji Y, Bennis M (2018) Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Internet Things J 6:4005–4018

Vemireddy S, Rout RR (2021) Fuzzy reinforcement learning for energy efficient task offloading in vehicular fog computing. Comput Netw 199:108463

Mousavi SS, Schukat M, Howley E (2018) Deep reinforcement learning: an overview. In: Proceedings of SAI intelligent systems conference (IntelliSys), 16

Mattner J, Lange S, Riedmiller M (2012) Learn to swing up and balance a real pole based on raw visual input data. In: International conference on neural information processing, pp 126–133

Levine S, Finn C, Darrell T, Abbeel P (2016) End-to-end training of deep visuomotor policies. J Mach Learn Res 17:1334–1373

MathSciNet   Google Scholar  

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518:529–533

Tuli S, Casale G, Jennings NR (2021) MCDS: AI augmented workflow scheduling in mobile edge cloud computing systems. IEEE Trans Parallel Distrib Syst 33:2794–2807

Mnih V, Badia A P, Mirza M, Graves A, Lillicrap TP, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: 33rd international conference on machine learning, vol 48, pp 1928–1937

Zolfpour-Arokhlo M, Selamat A, Hashim SZM, Afkhami H (2014) Modeling of route planning system based on Q value-based dynamic programming with multi-agent reinforcement learning algorithms. Eng Appl Artif Intell 29:163–177

Shi J, Du J, Wang J, Wang J, Yuan J (2020) Priority-aware task offloading in vehicular fog computing based on deep reinforcement learning. IEEE Trans Veh Technol 69:16067–16081

Chen M, Xiao Y, Li Q, Chen KC (2020) Minimizing age-of-information for fog computing-supported vehicular networks with deep Q-learning. In: IEEE international conference on communications (ICC), pp 1–6

Gazori P, Rahbari D, Nickray M (2020) Saving time and cost on the scheduling of fog-based IoT applications using deep reinforcement learning approach. Future Gener Comput Syst 110:1098–1115

Jamil B, Ijaz H, Shojafar M, Munir K, Buyya R (2022) Resource allocation and task scheduling in fog computing and internet of everything environments: a taxonomy, review, and future directions. ACM Comput Surv 54:1–38

Ghafari R, Hassani Kabutarkhani F, Mansouri N (2022) Task scheduling algorithms for energy optimization in cloud environment: a comprehensive review. Clust Comput 25:1035–1093

Wang M, Li Y, Zhang L, Pei F (2021) Research on intelligent workshop resource scheduling method based on improved NSGA-II algorithm. Robot Comput-Integr Manuf 71:102–141

Murad SA, Muzahid AJMd, Azmi ZRM, Hoque MdI, Kowsher Md (2022) A review on job scheduling technique in cloud computing and priority rule based intelligent framework. J King Saud Univ Comput Inf Sci 34:2309–2331

Dong T, Xue F, Xiao CH, Zhang J (2021) Workflow scheduling based on deep reinforcement learning in the cloud environment. J Ambient Intell Humaniz Comput 12:10823–10835

Xu R, Wang Y, Luo H, Wang F, Xie Y, Liu X, Yang Y (2018) A sufficient and necessary temporal violation handling point selection strategy in cloud workflow. Future Gener Comput Syst Int J ESci 86:464–479

Jalali Khalil Abadi Z, Mansouri N (2024) A comprehensive survey on scheduling algorithms using fuzzy systems in distributed environments. Artif Intell Rev 57(4)

Barut C, Yildirim G, Tatar Y (2024) An intelligent and interpretable rule-based metaheuristic approach to task scheduling in cloud systems. Knowl-Based Syst 284:111241

Zhang J, Guo B, Ding X, Hu D, Tang J, Du K, Tang C, Jiang Y (2024) An adaptive multi-objective multi-task scheduling method by hierarchical deep reinforcement learning. Appl Soft Comput 154:111342

Thilak KD, Devi KL, Shanmuganathan C, Kalaiselvi K (2024) Meta-heuristic algorithms to optimize two-stage task scheduling in the cloud. SN Comput Sci 5:122

Pradeep K, Gobalakrishnan N, Manikandan N, Javid Ali L, Parkavi K, Vijayakumar KP (2021) A review on task scheduling using optimization algorithm in clouds. In: 5th international conference on trends in electronics and informatics (ICOEI), pp 935–938

Vijayalakshmi V, Saravanan M (2022) An extensive analysis of task scheduling algorithms based on fog computing QoS metrics. In: International conference on innovative computing, intelligent communication and smart electrical systems (ICSES), pp 1–8

Walia NK, Kaur N (2021) Performance analysis of the task scheduling algorithms in the cloud computing environments. In: 2nd international conference on intelligent engineering and management (ICIEM), pp 108–113

Shyalika C, Silva T, Karunananda A (2020) Reinforcement learning in dynamic task scheduling: a review. SN Comput Sci 1:306

Wang H, Wang H (2022) Survey on task scheduling in cloud computing environment. artificial intelligent, Robot Hum-Comput Interact 286–291

Ciptaningtyas HT, Shiddiqi AM, Purwitasari D (2022) Survey on task scheduling methods in cloud RPS system. Int Semin Intell Technol Its Appl 151–156

George N, Nandhakumar KG, Vijayan VP (2021) Survival study on resource utilization and task scheduling in cloud. In: Second international conference on electronics and sustainable communication systems (ICESC), pp 1814–1819

Pol SS, Singh A (2021) Task scheduling algorithms in cloud computing: a survey. In: Second international conference on secure cyber computing and communication (ICSCCC), pp 244–249

Houssein EH, Gad AG, Wazery YM, Suganthan PN (2021) Task scheduling in cloud computing based on meta-heuristics: review, taxonomy, open challenges, and future trends. Swarm Evol Comput 62:100841

Archana R, Kumar PM (2022) Utilization of fog computing in task scheduling and offloading: modern growth and future challenges. In: International conference on electronic systems and intelligent computing (ICESIC), pp 23–28

Houa H, Jawaddia SNA, Ismail A (2024) Energy efficient task scheduling based on deep reinforcement learning in cloud environment: a specialized review. Future Gener Comput Syst 151:214–231

Sahni Y, Cao J, Yang L, Wang A (2022) Distributed resource scheduling in edge computing: problems, solutions, and opportunities. Comput Netw 219:109430

Wang Y, Yu J, Yu Z (2023) Resource scheduling techniques in cloud from a view of coordination: a holistic survey. Front Inf Technol Electron Eng 24:1–40

Lou Q, Hu S, Li C, Li G, Shi W (2021) Resource scheduling in edge computing: a survey. IEEE Commun Surv Tutor 23:2131–2165

Aron R, Abraham A (2022) Resource scheduling methods for cloud computing environment: the role of meta-heuristics and artificial intelligence. Eng Appl Artif Intell 116:105345

Rashidifar R, Bouzary H, Chen FF (2022) Resource scheduling in cloud based manufacturing system: a comprehensive survey. Int J Adv Manuf Technol 122:4201–4219

Rahimikhanghah A, Tajkey M, Rezazadeh B, Rahmani AM (2022) Resource scheduling methods in cloud and fog computing environments: a systematic literature review. Clust Comput 25:911–945

Himanshu, Mangla N (2021) Soft security resource scheduling issues in cloud computing: A review. In: 6th international conference on signal processing, computing and control (ISPCC), pp 678–684

Reddy KLR, Lathigara A, Aluvalu R (2021) Survey on load balancing techniques and resource scheduling in cloud computing. In: 4th smart cities symposium (SCS)

Rupali, Mangla N (2021) A critical review of workflow scheduling algorithms in cloud computing environment. In: Fourth international conference on computational intelligence and communication technologies (CCICT), pp 355–361

Kumar Y, Kaul S, Hu YC (2022) Machine learning for energy-resource allocation, workflow scheduling and live migration in cloud computing: State-of-the-art survey. Sustain Comput Inform Syst 36:100780

Singh G, Chaturvedi AK (2021) Particle swarm optimization-based approaches for cloud-based task and workflow scheduling: a systematic literature review. In: Second international conference on secure cyber computing and communication (ICSCCC), pp 350–358

Menaka M, Kumar KSS (2022) Workflow scheduling in cloud environment—Challenges, tools, limitations & methodologies: a review. Meas Sens 24:100436

Cao Z, Zhang H, Cao Y, Liu B (2019) A deep reinforcement learning approach to multi-component job scheduling in edge computing. In: 15th international conference on mobile ad-hoc and sensor networks (MSN), pp 19–24

Lu T, Zeng F, Shen J, Chen G, Shu W, Zhang W (2021) A scheduling scheme in a container-based edge computing environment using deep reinforcement learning approach. In: 17th international conference on mobility, sensing and networking (MSN), pp 56–65

Ju X, Su S, Xu C, Wang H (2023) Computation offloading and tasks scheduling for the internet of vehicles in edge computing: a deep reinforcement learning-based pointer network approach. Comput Netw 223:109572

Qi F, Zhou L, Xin C (2020) Deep reinforcement learning based task scheduling in edge computing networks. In: IEEE/CIC international conference on communications in China (ICCC), pp 835–840

Jayanetti A, Halgamuge S, Buyya R (2022) Deep reinforcement learning for energy and time optimized scheduling of precedence-constrained tasks in edge–cloud computing environments. Future Gener Comput Syst 137:14–30

Zhao Y, Li B, Wang J, Jiang D, Li D (2022) Integrating deep reinforcement learning with pointer networks for service request scheduling in edge computing. Knowl-Based Syst 258:109983

Zhang Y, Li R, Zhao Y, Li R, Wang Y, Zhou Z (2023) Multi-agent deep reinforcement learning for online request scheduling in edge cooperation networks. Future Gener Comput Syst 141:258–268

Meng H, Chao D, Huo R, Guo Q, Li X, Huang T (2019) Deep reinforcement learning based delay-sensitive task scheduling and resource management algorithm for multi-user mobile-edge computing systems. In: 4th international conference on mathematics and artificial intelligence, pp 66–70

Huang Y, Cheng L, Xue L, Liu C, Li Y, Li J, Ward T (2022) Deep adversarial imitation reinforcement learning for QoS-aware cloud job scheduling. IEEE Syst J 16:4232–4242

Yang Y, Shen H (2022) Deep reinforcement learning enhanced greedy optimization for online scheduling of batched tasks in cloud HPC systems. IEEE Trans Parallel Distrib Syst 33:3003–3014

Cheng M, Li J, Nazarian S (2018) DRL-cloud: deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. In: 23rd Asia and South Pacific design automation conference (ASP-DAC)

Yan J, Huang Y, Gupta A, Gupta A, Liu C, Li J, Cheng L (2022) Energy-aware systems for real-time job scheduling in cloud data centers: a deep reinforcement learning approach. Comput Electr Eng 99:107688

Tawfiqul Islam M, Karunasekera S, Buyya R (2022) Performance and cost-efficient spark job scheduling based on deep reinforcement learning in cloud computing environments. IEEE Trans Parallel Distrib Syst 33:1695–1710

Cheng L, Kalapgar A, Jain A, Wang Y, Qin Y, Li Y, Liu C (2022) Cost-aware real-time job scheduling for hybrid cloud using deep reinforcement learning. Neural Comput Appl 34:18579–18593

Cheng F, Huang Y, Tanpure B, Sawalani P, Cheng L, Liu C (2022) Cost-aware job scheduling for cloud instances using deep reinforcement learning. Clust Comput 25:619–631

Siddesha K, Jayaramaiah GV, Singh C (2022) A novel deep reinforcement learning scheme for task scheduling in cloud computing. Clust Comput 25:4171–4188

Ran L, Shi X, Shang M (2019) SLAs-aware online task scheduling based on deep reinforcement learning method in cloud environment. In: IEEE 21st international conference on high performance computing and communications, pp 1518–1525

Wang X, Zhang L, Liu Y, Zhao C, Wang K (2022) Solving task scheduling problems in cloud manufacturing via attention mechanism and deep reinforcement learning. J Manuf Syst 65:452–468

Swarup S, Shakshuki EM, Yasar A (2021) Task scheduling in cloud using deep reinforcement learning. Procedia Comput Sci 184:42–51

Rjoub G, Bentahar J, Abdel Wahab O, Bataineh AS (2020) Deep and reinforcement learning for automated task scheduling in large-scale cloud computing systems. Concurr Comput Pract Exp 33:1–14

Dong T, Xue F, Xiao C, Li J (2019) Task scheduling based on deep reinforcement learning in a cloud manufacturing environment. Concurr Comput Pract Exp 32:1–12

Mangalampalli S, Karri GR, Kumar M, Khalaf OI, Romero CAT, Abdul Sahib G (2024) DRLBTSA: deep reinforcement learning based task scheduling algorithm in cloud computing. Multimed Tools Appl 83:8359–8387

Swarup S, Shakshuki EM, Yasar A (2021) Energy efficient task scheduling in fog environment using deep reinforcement learning approach. Procedia Comput Sci 191:65–75

Beak J, Kaddoum G (2022) Online partial offloading and task scheduling in SDN-fog networks with deep recurrent reinforcement learning. IEEE Internet Things J 9:11578–11589

Ali Ibrahim M, Askar S (2023) An intelligent scheduling strategy in fog computing system based on multi-objective deep reinforcement learning algorithm. IEEE Access 11:133607–133622

Wang Z, Goudarzi M, Gong M, Buyya R (2024) Deep reinforcement learning-based scheduling for optimizing system load and response time in edge and fog computing environments. Future Gener Comput Syst 152:55–69

Xue F, Hai Q, Dong T, Cui Z, Gong Y (2022) A deep reinforcement learning based hybrid algorithm for efficient resource scheduling in edge computing environment. Inf Sci 608:362–374

Zhu H, Li M, Tang Y, Sun Y (2020) A deep-reinforcement-learning-based optimization approach for real-time scheduling in cloud manufacturing. IEEE Access 8:9987–9997

Zhou G, Wen R, Tian W, Buyya R (2022) Deep reinforcement learning-based algorithms selectors for the resource scheduling in hierarchical cloud computing. J Netw Comput Appl 208:103520

Karthik P, Sekhar K (2021) Resource scheduling approach in cloud testing as a service using deep reinforcement learning algorithms. CAAI Trans Intell Technol 147–154

Uma J, Vivekanandan P, Shankar S (2023) Optimized intellectual resource scheduling using deep reinforcement Q-learning in cloud computing. Trans Emerg Telecommun Technol 33:1–19

Long T, Xia Y, Ma Y, Peng Q, Zhao J (2022) A fault-tolerant workflow scheduling method on deep reinforcement learning-based in edge environment. In: IEEE international conference on networking, sensing and control (ICNSC)

Zheng T, Wan J, Zhang J, Jiang C (2022) Deep reinforcement learning-based workload scheduling for edge computing. J Cloud Comput Adv Syst Appl 11:1–13

Chen G, Qi J, Sun Y, Hu X, Dong Z, Sun Y (2023) A collaborative scheduling method for cloud computing heterogeneous workflows based on deep reinforcement learning. Future Gener Comput Syst 141:284–297

Dong T, Xue F, Xiao C (2021) Deep reinforcement learning for dynamic workflow scheduling in cloud environment. In: IEEE international conference on services computing (SCC), pp 107–115

Dong T, Xue F, Tang H, Xiao C (2022) Deep reinforcement learning for fault-tolerant workflow scheduling in cloud environment. Appl Intell 53:9916–9932

Li H, Huang J, Wang B, Fan Y (2022) Weighted double deep Q-network based reinforcement learning for bi-objective multi-workflow scheduling in the cloud. Clust Comput 25:751–768

Mangalampalli S, Hashemi SS, Gupta A, Rajkumar KV, Chakrabarti T, Chakrabarti P, Margala M (2024) Multi objective prioritized workflow scheduling using deep reinforcement based learning in cloud computing. IEEE Access 12:5373–5392

Zhang J, Cheng L, Liu C, Zhao Z, Mao Y (2023) Cost-aware scheduling systems for real-time workflows in cloud: an approach based on genetic algorithm and deep reinforcement learning. Expert Syst Appl 234:120972

https://pytorch.org/features/

Kumar R, Sahoo G (2014) Cloud computing simulation using CloudSim. Int J Eng Trends Technol 8:82–86

Chen W, Deelman E (2021) WorkflowSim: a toolkit for simulating scientific workflows in distributed environments. In: IEEE 8th international conference on E-science

da Silva RF, Chen W, Juve G, Vahi K, Deelman E (2014) Community resources for enabling research in distributed scientific workflows. In: IEEE 10th international conference on e-science

Silva Filho MC, Oliveira RL, Monteiro CC, Inácio PRM, Freire MM (2017) CloudSim Plus: a cloud computing simulation framework pursuing software engineering principles for improved modularity, extensibility and correctness. In: IFIP/IEEE symposium on integrated network and service management (IM)

Buyya R, Srirama SN (2019) Modelling and simulation of fog and edge computing environments using iFogSim toolkit. Wiley Telecom 433–465

Abreu DP, Velasquez K, Curado M, Monteiro E (2020) A comparative analysis of simulators for the cloud to Fog continuum. Simul Model Pract Theory 101:102029

Mahmud R, Pallewatta S, Goudarzi M, Buyya R (2022) iFogSim2: an extended iFogSim simulator for mobility, clustering, and microservice management in edge and fog computing environments. J Syst Softw 109:111351

Levine S, Kumar A, Tucker G, Fu J (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. Computer Science, Education

Download references

Author information

Authors and affiliations.

Department of Computer Science, Shahid Bahonar University of Kerman, Box No. 76135-133, Kerman, Iran

Zahra Jalali Khalil Abadi, Najme Mansouri & Mohammad Masoud Javidi

You can also search for this author in PubMed   Google Scholar

Contributions

Zahra Jalali Khalil Abadi designed the study, gathered the data for the article, and participated in writing—original draft preparation. Najme Mansouri contributed to the investigation, interpretation of the results, and writing—original draft preparation. Mohammad Masoud Javidi was involved in the verification, writing—reviewing and editing.

Corresponding author

Correspondence to Najme Mansouri .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Ethics approval and consent to participate

Consent for publication, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Jalali Khalil Abadi, Z., Mansouri, N. & Javidi, M.M. Deep reinforcement learning-based scheduling in distributed systems: a critical review. Knowl Inf Syst (2024). https://doi.org/10.1007/s10115-024-02167-7

Download citation

Received : 15 August 2023

Revised : 07 May 2024

Accepted : 16 June 2024

Published : 26 June 2024

DOI : https://doi.org/10.1007/s10115-024-02167-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Deep reinforcement learning
• Cloud computing
• Fog computing
• Find a journal
• Publish with us
• Track your research