research papers credibility

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Am J Pharm Educ
• v.84(1); 2020 Jan

A Review of the Quality Indicators of Rigor in Qualitative Research

Jessica l. johnson.

a William Carey University School of Pharmacy, Biloxi, Mississippi

Donna Adkins

Sheila chauvin.

b Louisiana State University, School of Medicine, New Orleans, Louisiana

Attributes of rigor and quality and suggested best practices for qualitative research design as they relate to the steps of designing, conducting, and reporting qualitative research in health professions educational scholarship are presented. A research question must be clear and focused and supported by a strong conceptual framework, both of which contribute to the selection of appropriate research methods that enhance trustworthiness and minimize researcher bias inherent in qualitative methodologies. Qualitative data collection and analyses are often modified through an iterative approach to answering the research question. Researcher reflexivity, essentially a researcher’s insight into their own biases and rationale for decision-making as the study progresses, is critical to rigor. This article reviews common standards of rigor, quality scholarship criteria, and best practices for qualitative research from design through dissemination.

INTRODUCTION

Within the past 20 years, qualitative research in health professions education has increased significantly, both in practice and publication. Today, one can pick up most any issue of a wide variety of health professions education journals and find at least one article that includes some type of qualitative research, whether a full study or the inclusion of a qualitative component within a quantitative or mixed methods study. Simultaneously, there have been recurrent calls for enhancing rigor and quality in qualitative research.

As members of the academic community, we share responsibility for ensuring rigor in qualitative research, whether as researchers who design and implement, manuscript reviewers who critique, colleagues who discuss and learn from each other, or scholarly teachers who draw upon results to enhance and innovate education. Therefore, the purpose of this article is to summarize standards of rigor and suggested best practices for designing, conducting, and reporting high-quality qualitative research. To begin, Denzin and Lincoln’s definition of qualitative research, a long-standing cornerstone in the field, provides a useful foundation for summarizing quality standards and best practices:

Qualitative research involves the studied use and collection of a variety of empirical materials – case study; personal experience; introspection; life story; interview; artifacts; cultural texts and productions; observational, historical, interactional, and visual texts – that describe the routine and problematic moments and meanings in individual lives. Accordingly, qualitative researchers deploy a wide range of interconnected interpretative practices, hoping always to get a better understanding of the subject matter at hand. It is understood, however, that each practice makes the world visible in a different way. Hence there is frequently a commitment to using more than one interpretative practice in any study. 1

In recent years, multiple publications have synthesized quality criteria and recommendations for use by researchers and peer reviewers alike, often in the form of checklists. 2-6 Some authors have raised concerns about the use of such checklists and adherence to strict, universal criteria because they do not afford sufficient flexibility to accommodate the diverse approaches and multiple interpretive practices often represented in qualitative studies. 7-11 They argue that a strict focus on using checklists of specific technical criteria may stifle the diversity and multiplicity of practices that are so much a part of achieving quality and rigor within the qualitative paradigm. As an alternative, some of these authors have published best practice guidelines for use by researchers and peer reviewers to achieve and assess methodological rigor and research quality. 12,13

Some journals within the field of health professions education have also established best practice guidance, as opposed to strict criteria or a checklist, for qualitative research. These have been disseminated as guiding questions or evaluation categories. In 2015, Academic Medicine produced an expanded second edition of a researcher/author manual that includes specific criteria with extensive explanations and examples. 14 Still others have disseminated best practice guidelines through a series of methodological articles within journal publications. 2

In this article, attributes of rigor and quality and suggested best practices are presented as they relate to the steps of designing, conducting, and reporting qualitative research in a step-wise approach.

BEST PRACTICES: STEP-WISE APPROACH

Step 1: identifying a research topic.

Identifying and developing a research topic is comprised of two major tasks: formulating a research question, and developing a conceptual framework to support the study. Formulating a research question is often stimulated by real-life observations, experiences, or events in the researcher’s local setting that reflect a perplexing problem begging for systematic inquiry. The research question begins as a problem statement or set of propositions that describe the relationship among certain concepts, behaviors, or experiences. Agee 15 and others 16,17 note that initial questions are usually too broad in focus and too vague regarding the specific context of the study to be answerable and researchable. Creswell reminds us that initial qualitative research questions guide inquiry, but they often change as the author’s understanding of the issue develops throughout the study. 16 Developing and refining a primary research question focused on both the phenomena of interest and the context in which it is situated is essential to research rigor and quality.

Glassick, Huber, and Maeroff identified six criteria applicable to assessing the quality of scholarship. 18,19 Now commonly referred to as the Glassick Criteria ( Table 1 ), these critical attributes outline the essential elements of any scholarly approach and serve as a general research framework for developing research questions and designing studies. The first two criteria, clear purpose and adequate preparation, are directly related to formulating effective research questions and a strong conceptual framework.

Glassick’s Criteria for Assessing the Quality of Scholarship of a Research Study 18

Generating and refining a qualitative research question requires thorough, systematic, and iterative review of the literature, and the use of those results to establish a clear context and foundation for the question and study design. Using an iterative approach, relevant concepts, principles, theories or models, and prior evidence are identified to establish what is known, and more importantly, what is not known. The iterative process contributes to forming a better research question, the criteria for which can be abbreviated by the acronym FINER, ie, f easible, i nteresting, n ovel, e thical, and r elevant, that is answerable and researchable, in terms of research focus, context specificity, and the availability of time, logistics, and resources to carry out the study. Developing a FINER research question is critical to study rigor and quality and should not be rushed, as all other aspects of research design depend on the focus and clarity of the research question(s) guiding the study. 15 Agee provides clear and worthwhile additional guidance for developing qualitative research questions. 15

Reflexivity, the idea that a researcher’s preconceptions and biases can influence decisions and actions throughout qualitative research activities, is a critical aspect of rigor even at the earliest stages of the study. A researcher’s background, beliefs, and experiences may affect any aspect of the research from choosing which specific question to investigate through determining how to present the results. Therefore, even at this early stage, the potential effect of researcher bias and any ethical considerations should be acknowledged and addressed. That is, how will the question’s influence on study design affect participants’ lives, position the researcher in relationship with others, or require specific methods for addressing potential areas of research bias and ethical considerations?

A conceptual framework is then actively constructed to provide a logical and convincing argument for the research. The framework defines and justifies the research question, the methodology selected to answer that question, and the perspectives from which interpretation of results and conclusions will be made. 5,6,20 Developing a well-integrated conceptual framework is essential to establishing a research topic based upon a thorough and integrated review of relevant literature (addressing Glassick criteria #1 and #2: clear purpose and adequate preparation). Key concepts, principles, assumptions, best practices, and theories are identified, defined, and integrated in ways that clearly demonstrate the problem statement and corresponding research question are answerable, researchable, and important to advancing thinking and practice.

Ringsted, Hodges, and Sherpbier describe three essential parts to an effective conceptual framework: theories and/or concepts and principles relevant to the phenomenon of interest; what is known and unknown from prior work, observations, and examples; and the researcher’s observations, ideas, and suppositions regarding the research problem statement and question. 21 Lingard describes four types of unknowns to pursue during literature review: what no one knows; what is not yet well understood; what controversy or conflicting results, understandings, or perspectives exist; and what are unproven assumptions. 22 In qualitative research, these unknowns are critical to achieving a well-developed conceptual framework and a corresponding rigorous study design.

Recent contributions from Ravitch and colleagues present best practices in developing frameworks for conceptual and methodological coherence within a study design, regardless of the research approach. 23,24 Their recommendations and arguments are highly relevant to qualitative research. Figure 1 reflects the primary components of a conceptual framework adapted from Ravitch and Carl 23 and how all components contribute to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice. Notice that each element of the framework interacts with and influences other elements in a dynamic and interactive process from the beginning to the end of a research project. The intersecting bidirectional arrows represent direct relationships between elements as they relate to specific aspects of a qualitative research study.

Adaptation of Ravitch and Carl’s Components of a Conceptual Framework 23

Maxwell also provides useful guidance for developing an effective conceptual framework specific to the qualitative research paradigm. 17 The 2015 second edition of the Review Criteria for Research Manuscripts 14 and work by Ravitch and colleagues 23,24 provide specific guidance for applying the conceptual framework to each stage of the research process to enhance rigor and quality. Quality criteria for assessing a study’s problem statement, conceptual framework, and research question include the following: introduction builds a logical case and provides context for the problem statement; problem statement is clear and well-articulated; conceptual framework is explicit and justified; research purpose and/or question is clearly stated; and constructs being investigated are clearly identified and presented. 14,24,25 As best practice guidelines, these criteria facilitate quality and rigor while providing sufficient flexibility in how each is achieved and demonstrated.

While a conceptual framework is important to rigor in qualitative research, Huberman and Miles caution qualitative researchers about developing and using a framework to the extent that it influences qualitative design deductively because this would violate the very principles of induction that define the qualitative research paradigm. 25 Our profession’s recent emphasis on a holistic admissions process for pharmacy students provides a reasonable example of inductive and deductive reasoning and their respective applications in qualitative and quantitative research studies. Principles of inductive reasoning are applied when a qualitative research study examines a representative group of competent pharmacy professionals to generate a theory about essential cognitive and affective skills for patient-centered care. Deductive reasoning could then be applied to design a hypothesis-driven prospective study that compares the outcomes of two cohorts of students, one group admitted using traditional criteria and one admitted based on a holistic admissions process revised to value the affective skills of applicants. Essentially, the qualitative researcher must carefully generate a conceptual framework that guides the research question and study design without allowing the conceptual framework to become so rigid as to dictate a testable hypothesis, which is the founding principle of deductive reasoning. 26

Step 2: Qualitative Study Design

The development of a strong conceptual framework facilitates selection of appropriate study methods to minimize the bias inherent in qualitative studies and help readers to trust the research and the researcher (see Glassick criteria #3 in Table 1 ). Although researchers can employ great flexibility in the selection of study methods, inclusion of best practice methods for assuring the rigor and trustworthiness of results is critical to study design. Lincoln and Guba outline four criteria for establishing the overall trustworthiness of qualitative research results: credibility, the researcher ensures and imparts to the reader supporting evidence that the results accurately represent what was studied; transferability, the researcher provides detailed contextual information such that readers can determine whether the results are applicable to their or other situations; dependability, the researcher describes the study process in sufficient detail that the work could be repeated; confirmability, the researcher ensures and communicates to the reader that the results are based on and reflective of the information gathered from the participants and not the interpretations or bias of the researcher. 27

Specific best practice methods used in the sampling and data collection processes to increase the rigor and trustworthiness of qualitative research include: clear rationale for sampling design decisions, determination of data saturation, ethics in research design, member checking, prolonged engagement with and persistent observation of study participants, and triangulation of data sources. 28

Qualitative research is focused on making sense of lived, observed phenomenon in a specific context with specifically selected individuals, rather than attempting to generalize from sample to population. Therefore, sampling design in qualitative research is not random but defined purposively to include the most appropriate participants in the most appropriate context for answering the research question. Qualitative researchers recognize that certain participants are more likely to be “rich” with data or insight than others, and therefore, more relevant and useful in achieving the research purpose and answering the question at hand. The conceptual framework contributes directly to determining sample definitions, size, and recruitment of participants. A typical best practice is purposive sampling methods, and when appropriate, convenience sampling may be justified. 29

Purposive sampling reflects intentional selection of research participants to optimize data sources for answering the research question. For example, the research question may be best answered by persons who have particular experience (critical case sampling) or certain expertise (key informant sampling). Similarly, additional participants may be referred for participation by active participants (snowball sampling) or may be selected to represent either similar or opposing viewpoints (confirming or disconfirming samples). Again, the process of developing and using a strong conceptual framework to guide and justify methodological decisions, in this case defining and establishing the study sample, is critical to rigor and quality. 30 Convenience sampling, using the most accessible research participants, is the least rigorous approach to defining a study sample and may result in low accuracy, poor representativeness, low credibility, and lack of transferability of study results.

Qualitative studies typically reflect designs in which data collection and analysis are done concurrently, with results of ongoing analysis informing continuing data collection. Determination of a final sample size is largely based on having sufficient opportunity to collect relevant data until new information is no longer emerging from data collection, new coding is not feasible, and/or no new themes are emerging; that is, reaching data saturation , a common standard of rigor for data collection in qualitative studies . Thus, accurately predicting a sample size during the planning phases of qualitative research can be challenging. 30 Care should be taken that sufficient quantity (think thick description) and quality (think rich description) of data have been collected prior to concluding that data saturation has been achieved. A poor decision regarding sample size is a direct consequence of sampling strategy and quality of data generated, which leaves the researcher unable to fully answer the research question in sufficient depth. 30

Though data saturation is probably the most common terminology used to describe the achievement of sufficient sample size, it does not apply to all study designs. For example, one could argue that in some approaches to qualitative research, data collection could continue infinitely if the event continues infinitely. In education, we often anecdotally observe variations in the personality and structure of a class of students, and as generations of students continue to evolve with time, so too would the data generated from observing each successive class. In such situations, data saturation might never be achieved. Conversely, the number of participants available for inclusion in a sample may be small and some risk of not reaching data saturation may be unavoidable. Thus, the idea of fully achieving data saturation may be unrealistic when applied to some populations or research questions. In other instances, attrition and factors related to time and resources may contribute to not reaching data saturation within the limits of the study. By being transparent in the process and reporting of results when saturation may not have been possible, the resulting data may still contribute to the field and to further inquiry. Replication of the study using other samples and conducting additional types of follow-up studies are other options for better understanding the research phenomenon at hand. 31

In addition to defining the sample and selecting participants, other considerations related to sampling bias may impact the quantity and quality of data generated and therefore the quality of the study result. These include: methods of recruiting, procedures for informed consent, timing of the interviews in relation to experience or emotion, procedures for ensuring participant anonymity/confidentiality, interview setting, and methods of recording/transcribing the data. Any of these factors could potentially change the nature of the relationship between the researcher and the study participants and influence the trustworthiness of data collected or the study result. Thus, ongoing application of previously mentioned researcher reflexivity is critical to the rigor of the study and quality of sampling. 29,30

Common qualitative data collection methods used in health professions education include interview, direct observation methods, and textual/document analysis. Given the unique and often highly sensitive nature of data being collected by the researcher, trustworthiness is an essential component of the researcher-participant relationship. Ethical conduct refers to how moral principles and values are part of the research process. Participants’ perceptions of ethical conduct are fundamental to a relationship likely to generate high quality data. During each step of the research process, care must be taken to protect the confidentiality of participants and shield them from harm relating to issues of respect and dignity. Researchers must be respectful of the participants’ contributions and quotes, and results must be reported truthfully and honestly. 8

Interview methods range from highly structured to increase dependability or completely open-ended to allow for interviewers to clarify a participant’s response for increased credibility and confirmability. Regardless, interview protocols and structure are often modified or refined, based on concurrent data collection and analysis processes to support or refute preliminary interpretations and refine focus and continuing inquiry. Researcher reflexivity, or acknowledgement of researcher bias, is absolutely critical to the credibility and trustworthiness of data collection and analysis in such study designs. 32

Interviews should be recorded and transcribed verbatim prior to coding and analysis. 28 Member checking, a common standard of rigor, is a practice to increase study credibility and confirmability that involves asking a research subject to verify the transcription of an interview. 1,16,28 The research subject is asked to verify the completeness and accuracy of an interview transcript to ensure the transcript truthfully reflects the meaning and intent of the subject’s contribution.

Prolonged engagement involves the researcher gaining familiarity and understanding of the culture and context surrounding the persons or situations being studied. This strategy supports reflexivity, allowing the researcher to determine how they themselves may be a source of bias during the data collection process by altering the nature of how individuals behave or interact with others in the presence of the researcher. Facial expressions, spoken language, body language, style of dress, age, race, gender, social status, culture, and the researcher’s relationship with the participants may potentially influence either participants’ responses or how the researcher interprets those responses. 33 “Fitting in” by demonstrating an appreciation and understanding of the cultural norms of the population being studied potentially allows the researcher to obtain more open and honest responses from participants. However, if the research participants or topic are too familiar or personal, this may also influence data collection or analysis and interpretation of the results. 33 The possible applications of this section to faculty research with student participants in the context of pharmacy education are obvious, and researcher reflexivity is critical to rigor.

Some researchers using observational methods adopt a strategy of direct field observation, while others play partial or full participant roles in the activity being observed. In both observation scenarios, it is impossible to separate the researcher from the environment, and researcher reflexivity is essential. The pros and cons of observation approach, relative to the research question and study purpose, should be evaluated by the researcher, and the justification for the observational strategy selected should be made clear. 34 Regardless of the researcher’s degree of visibility to the study participants, persistent observation of the targeted sample is critical to the confirmability standard and to achieving data saturation. That is, study conclusions must be clearly grounded in persistent phenomena witnessed during the study, rather than on a fluke event. 28

Researchers acknowledge that observational methodologies are limited by the reality that the researcher carries a bias in determining what is observed, what is recorded, how it is recorded, and how it is transcribed for analysis. A study’s conceptual framework is critical to achieving rigor and quality and provides guidance in developing predetermined notions or plans for what to observe, how to record, and how to minimize the influence of potential bias. 34 Researcher notes should be recorded as soon as possible after the observation event to optimize accuracy. The more detailed and complete the notes, the more accurate and useful they can be in data analysis or in auditing processes for enhancing rigor in the interpretation phase of the study. 34

Triangulation is among the common standards of rigor applied within the qualitative research paradigm. Data triangulation is used to identify convergence of data obtained through multiple data sources and methods (eg, observation field notes and interview transcripts) to avoid or minimize error or bias and optimize accuracy in data collection and analysis processes. 33,35,36

Again, researcher practice in reflexivity throughout research processes is integral to rigor in study design and implementation. Researchers must demonstrate attention to appropriate methods and reflective critique, which are represented in both core elements of the conceptual framework ( Figure 1 ) and Glassick criteria ( Table 1 ). In so doing, the researcher will be well-prepared to justify sampling design and data collection decisions to manuscript reviewers and, ultimately, readers.

Step 3: Data Analysis

In many qualitative studies, data collection runs concurrently with data analysis. Specific standards of rigor are commonly used to ensure trustworthiness and integrity within the data analysis process, including use of computer software, peer review, audit trail, triangulation, and negative case analysis.

Management and analyses of qualitative data from written text, observational field notes, and interview transcriptions may be accomplished using manual methods or the assistance of computer software applications for coding and analysis. When managing very large data sets or complex study designs, computer software can be very helpful to assist researchers in coding, sorting, organizing, and weighting data elements. Software applications can facilitate ease in calculating semi-quantitative descriptive statistics, such as counts of specific events, that can be used as evidence that the researcher’s analysis is based on a representative majority of data collected ( inclusivism ) rather than focusing on selected rarities ( anecdotalism ). Using software to code data can also make it easier to identify deviant cases, detect coding errors, and estimate interrater reliability among multiple coders. 37 While such software helps to manage data, the actual analyses and interpretation still reside with the researcher.

Peer review, another common standard of rigor, is a process by which researchers invite an independent third-party researcher to analyze a detailed audit trail maintained by the study author. The audit trail methodically describes the step-by-step processes and decision-making throughout the study. Review of this audit trail occurs prior to manuscript development and enhances study confirmability. 1,16 The peer reviewer offers a critique of the study methods and validation of the conclusions drawn by the author as a thorough check on researcher bias.

Triangulation also plays a role in data analysis, as the term can also be used to describe how multiple sources of data can be used to confirm or refute interpretation, assertions, themes, and study conclusions. If a theme or theory can be arrived at and validated using multiple sources of data, the result of the study has greater credibility and confirmability. 16,33,36 Should any competing or controversial theories emerge during data collection or analysis, it is vital to the credibility and trustworthiness of the study that the author disclose and explore those negative cases. Negative case analysis refers to actively seeking out and scrutinizing data that do not fit or support the researcher’s interpretation of the data. 16

The use of best practices applying to data collection and data analysis facilitates the full examination of data relative to the study purpose and research question and helps to prevent premature closure of the study. Rather than stopping at the initial identification of literal, first-level assertion statements and themes, authors must progress to interpreting how results relate to, revise, or expand the conceptual framework, or offer an improved theory or model for explaining the study phenomenon of interest. Closing the loop on data collection is critical and is achieved when thorough and valid analysis can be linked back to the conceptual framework, as addressed in the next section.

Step 4: Drawing Valid Conclusions

Lingard and Kennedy 38 succinctly state that the purpose of qualitative research is to deepen one’s understanding of specific perspectives, observations, experiences, or events evidenced through the behaviors or products of individuals and groups as they are situated in specific contexts or circumstances. Conclusions generated from study results should enhance the conceptual framework, or contribute to a new theory or model development, and are most often situated within the discussion and conclusion sections of a manuscript.

The discussion section should include interpretation of the results and recommendations for practice. Interpretations should go beyond first-level results or literal description of observed behaviors, patterns, and themes from analysis. The author’s challenge is to provide a complete and thorough examination and explanation of how specific results relate to each other, contribute to answering the research question, and achieve the primary purpose of the research endeavor. The discussion should “close the loop” by integrating study results and analysis with the original conceptual framework. The discussion section should also provide a parsimonious narrative or graphical explanation and interpretation of study results that enhances understanding of the targeted phenomena.

The conclusion section should provide an overall picture or synopsis of the study, including its important and unique contributions to the field from the perspective of both conceptual and practical significance. The conclusion should also include personal and theoretical perspectives and future directions for research. Together, the discussion and conclusion should include responses to the larger questions of the study’s contributions, such as: So what? Why do these results matter? What next?

The strength of conclusions is dependent upon the extent to which standards of rigor and best practices were demonstrated in design, data collection, data analysis, and interpretation, as described in previous sections of this article. 4,12,17,23,24 Quality and rigor expectations for drawing valid conclusions and generating new theories are reflected in the following essential features of rigor and quality, which include: “Close the loop” to clearly link research questions, study design, data collection and analysis, and interpretation of results. Reflect effective integration of the study results with the conceptual framework and explain results in ways that relate, support, elaborate, and/or challenge conclusions of prior scholarship. Descriptions of new or enhanced frameworks or models are clear and effectively grounded in the study results and conclusions. Practical or theoretical implications are effectively discussed, including guidance for future studies. Limitations and issues of reflexivity and ethics are clearly and explicitly described, including references to actions taken to address these areas. 3,4,12,14

Step 5: Reporting Research Results

Key to quality reporting of qualitative research results are clarity, organization, completeness, accuracy, and conciseness in communicating the results to the reader of the research manuscript. O’Brien and others 4 proposed a standardized framework specifically for reporting qualitative studies known as the Standards for Reporting Qualitative Research (SRQR, Table 2 ). This framework provides detailed explanations of what should be reported in each of 21 sections of a qualitative research manuscript. While the SRQR does not explicitly mention a conceptual framework, the descriptions and table footnote clarification for the introduction and problem statement reflect the essential elements and focus of a conceptual framework. Ultimately, readers of published work determine levels of credibility, trustworthiness, and the like. A manuscript reviewer, the first reader of a study report, has the responsibility and privilege of providing critique and guidance to authors regarding achievement of quality criteria, execution and reporting of standards of rigor, and the extent to which meaningful contributions to thinking and practice in the field are presented. 13,39

An Adaptation of the 21 Elements of O’Brien and Colleagues’ Standards for Reporting Qualitative Research (SRQR) 4

Authors must avoid language heavy with connotations or adjectives that insert the researcher’s opinion into the database or manuscript. 14,40 The researcher should be as neutral and objective as possible in interpreting data and in presenting results. Thick and rich descriptions, where robust descriptive language is used to provide sufficient contextual information, enable the reader to determine credibility, transferability, dependability, and confirmability .

The process of demonstrating the credibility of research is rooted in honest and transparent reporting of how biases and other possible confounders were identified and addressed throughout study processes. Such reporting, first described within the study’s conceptual framework, should be revisited in reporting the work. Confounders may include the researcher’s training and previous experiences, personal connections to the background theory, access to the study population, and funding sources. These elements and processes are best represented in Glassick’s criteria for effective presentation and reflective critique ( Table 1 , criteria 5 and 6). Transferability is communicated, in part, through description of sampling factors such as: geographical location of the study, number and characteristics of participants, and the timeframe of data collection and analysis. 40 Such descriptions also contribute to the credibility of the results and readers’ determination of transfer to their and other contexts. To ensure dependability, the research method must be reported in detail such that the reader can determine proper research practices have been followed and that future researchers can repeat the study. 40 The confirmability of the results is influenced by reducing or at a minimum explaining any researcher influence on the result by applying and meeting standards of rigor such as member checking, triangulation, and peer review. 29,33

In qualitative studies, the researcher is often the primary instrument for data collection. Any researcher biases not adequately addressed or errors in judgement can affect the quality of data and subsequent research results. 33 Thus, due to the creative interpretative and contextually bound nature of qualitative studies, the application of standards of rigor and adherence to systematic processes well-documented in an audit trail are essential. The application of rigor and quality criteria extend beyond the researcher and are also important to effective peer review processes within a study and for scholarly dissemination. The goal of rigor in qualitative research can be described as ensuring that the research design, method, and conclusions are explicit, public, replicable, open to critique, and free of bias. 41 Rigor in the research process and results are achieved when each element of study methodology is systematic and transparent through complete, methodical, and accurate reporting. 33 Beginning the study with a well-developed conceptual framework and active use of both researcher reflexivity and rigorous peer review during study implementation can drive both study rigor and quality.

As the number of published qualitative studies in health professions educational research increases, it is important for our community of health care educators to keep in mind the unique aspects of rigor in qualitative studies presented here. Qualitative researchers should select and apply any of the above referenced study methods and research practices, as appropriate to the research question, to achieve rigor and quality. As in any research paradigm, the goal of quality and rigor in qualitative research is to minimize the risk of bias and maximize the accuracy and credibility of research results. Rigor is best achieved through thoughtful and deliberate planning, diligent and ongoing application of researcher reflexivity, and honest communication between the researcher and the audience regarding the study and its results.

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

• PLOS Biology
• PLOS Climate
• PLOS Complex Systems
• PLOS Computational Biology
• PLOS Digital Health
• PLOS Genetics
• PLOS Global Public Health
• PLOS Medicine
• PLOS Mental Health
• PLOS Neglected Tropical Diseases
• PLOS Pathogens
• PLOS Sustainability and Transformation
• PLOS Collections
• About This Blog
• Official PLOS Blog
• EveryONE Blog
• Speaking of Medicine
• PLOS Biologue
• Absolutely Maybe
• DNA Science
• PLOS ECR Community
• All Models Are Wrong
• About PLOS Blogs

Impact AND credibility matter when researchers evaluate research

by Veronique Kiermer, Iain Hrynaszkiewicz, & James Harney.

Today we’ve posted a report , along with accompanying data, on qualitative research we conducted about how researchers assess the credibility and impact of research. This study, which has not yet been peer reviewed, was supported by a grant from the Alfred P. Sloan Foundation and conducted with the assistance of the American Society for Cell Biology . The findings will inform future PLOS activities to support improved research assessment practices — specifically to support efforts to move emphasis towards individual research outputs and away from journal-level metrics. 

As we wrote in October 2020, we are interested in how researchers evaluate research outputs when (1) conducting their own research, and (2) when they take part in committees for hiring or grant review. In particular, we were interested in how researchers make judgments about the credibility and impact of the research outputs — including papers, preprints, research data — that they encounter in these contexts.

We interviewed 52 cell biology researchers.  Our approach focused on the goals they are trying to achieve (e.g.”identify impactful research to read”), rather than the tools they are presently using to carry out these tasks. By focusing on researchers’ goals (the what ) rather than how they are achieving them, we sought to better understand how we might influence those practices. This qualitative research will be followed by survey work to better quantify our findings. This will provide insights into opportunities for better solutions for improved research assessment. In particular, we’ll understand what signals of credibility and impact might provide researchers with more useful ways than journal impact factor or journal prestige to assess the quality and credibility of individual studies and individual researchers.

Our results confirmed our initial hypothesis that the credibility (or trustworthiness) of research outputs is the central concern for researchers when conducting their own research, and that impact was a strong focus when researchers are part of hiring or grant review committees. But we established that researchers also assess attributes of research outputs related to reproducibility, quality, and novelty. 

In addition, we found that researchers said they assessed credibility in committees more frequently than we anticipated, given that impact considerations — including journal impact factor — are prevalent in committee guidance and research assessment objectives (see for example McKiernan et al . (2019), Niles et al. (2020), Alperin et al. (2020), and Sugimoto & Larivière (2018)).

Our interviews confirmed that convenient proxies for credibility and impact, usually those based on journals, are used pervasively and are common in both research discovery and committee activities. 

Our research also indicates that when researchers inspect publications to evaluate credibility they try to minimize the amount of time they spend reading and understanding publications. Their tactics included selective reading of the abstracts, figures, and methods sections. Sometimes they said that they also look for signals such as whether data was available and had been reused, whether peer-reviewed versions of preprints have been published, and whether open peer review reports were available. 

Insights that help us better understand what researchers’ goals are and how they make judgements about credibility when discovering and reading research may offer opportunities to provide more reliable signals that help them with these tasks, yet are better tailored for credibility judgments than journal-level metrics. The stated importance of assessing credibility by researchers who participate in research assessment committees also suggests an opportunity for funders and institutions to better align their guidelines with the practice and motivations of committee members. 

After our follow-up survey work to validate these preliminary findings, we will report back and hope that this research will help others in the understanding and development of better methods of research assessment.

Author: Iain Hrynaszkiewicz, Director, Open Research Solutions, PLOS; and Chris Heid, Head of Product, PLOS Summary Research by PLOS and Research Consulting…

We’re pleased to once again annual reporting from the Plan S Price & Service Transparency Framework for 2022…

Written by Lindsay Morton Over 4 years: 74k+ eligible articles. Nearly 85k signed reviews. More than 30k published peer review history packages…

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 18, Issue 2
• Issues of validity and reliability in qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Helen Noble 1 ,
• Joanna Smith 2
• 1 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• 2 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
• Correspondence to Dr Helen Noble School of Nursing and Midwifery, Queens's University Belfast, Medical Biology Centre, 97 Lisburn Rd, Belfast BT9 7BL, UK; helen.noble{at}qub.ac.uk

https://doi.org/10.1136/eb-2015-102054

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Evaluating the quality of research is essential if findings are to be utilised in practice and incorporated into care delivery. In a previous article we explored ‘bias’ across research designs and outlined strategies to minimise bias. 1 The aim of this article is to further outline rigour, or the integrity in which a study is conducted, and ensure the credibility of findings in relation to qualitative research. Concepts such as reliability, validity and generalisability typically associated with quantitative research and alternative terminology will be compared in relation to their application to qualitative research. In addition, some of the strategies adopted by qualitative researchers to enhance the credibility of their research are outlined.

Are the terms reliability and validity relevant to ensuring credibility in qualitative research?

Although the tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research, there are ongoing debates about whether terms such as validity, reliability and generalisability are appropriate to evaluate qualitative research. 2–4 In the broadest context these terms are applicable, with validity referring to the integrity and application of the methods undertaken and the precision in which the findings accurately reflect the data, while reliability describes consistency within the employed analytical procedures. 4 However, if qualitative methods are inherently different from quantitative methods in terms of philosophical positions and purpose, then alterative frameworks for establishing rigour are appropriate. 3 Lincoln and Guba 5 offer alternative criteria for demonstrating rigour within qualitative research namely truth value, consistency and neutrality and applicability. Table 1 outlines the differences in terminology and criteria used to evaluate qualitative research.

• View inline

Terminology and criteria used to evaluate the credibility of research findings

What strategies can qualitative researchers adopt to ensure the credibility of the study findings?

Unlike quantitative researchers, who apply statistical methods for establishing validity and reliability of research findings, qualitative researchers aim to design and incorporate methodological strategies to ensure the ‘trustworthiness’ of the findings. Such strategies include:

Accounting for personal biases which may have influenced findings; 6

Acknowledging biases in sampling and ongoing critical reflection of methods to ensure sufficient depth and relevance of data collection and analysis; 3

Meticulous record keeping, demonstrating a clear decision trail and ensuring interpretations of data are consistent and transparent; 3 , 4

Establishing a comparison case/seeking out similarities and differences across accounts to ensure different perspectives are represented; 6 , 7

Including rich and thick verbatim descriptions of participants’ accounts to support findings; 7

Demonstrating clarity in terms of thought processes during data analysis and subsequent interpretations 3 ;

Engaging with other researchers to reduce research bias; 3

Respondent validation: includes inviting participants to comment on the interview transcript and whether the final themes and concepts created adequately reflect the phenomena being investigated; 4

Data triangulation, 3 , 4 whereby different methods and perspectives help produce a more comprehensive set of findings. 8 , 9

Table 2 provides some specific examples of how some of these strategies were utilised to ensure rigour in a study that explored the impact of being a family carer to patients with stage 5 chronic kidney disease managed without dialysis. 10

Strategies for enhancing the credibility of qualitative research

In summary, it is imperative that all qualitative researchers incorporate strategies to enhance the credibility of a study during research design and implementation. Although there is no universally accepted terminology and criteria used to evaluate qualitative research, we have briefly outlined some of the strategies that can enhance the credibility of study findings.

• Sandelowski M
• Lincoln YS ,
• Barrett M ,
• Mayan M , et al
• Greenhalgh T
• Lingard L ,

Twitter Follow Joanna Smith at @josmith175 and Helen Noble at @helnoble

Competing interests None.

Read the full text or download the PDF:

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

2.7: Evaluating the Quality and Credibility of Your Research

• Last updated
• Save as PDF
• Page ID 6613

• Steven D. Krause
• Eastern Michigan University

Finding evidence that answers a question is only the first part of the research process. You also have to evaluate the quality and credibility of your research. Inevitably, as we’ve already seen in this chapter, you do this as you consider the origins of your research—primary versus secondary research, scholarly versus popular sources, the Internet, and so forth. But evaluating the quality and credibility of your research is more subtle and complicated than just determining the source of the evidence. Consider again the example from the beginning of this chapter about deciding which computer to buy. One of the things you would have to weigh is the credibility of the information you received from your friends compared to the information you received from a salesperson at the computer store. You can probably count on your friends to be trustworthy and honest, but they might not know much about computers. Conversely, while a salesperson might know a lot about computers, you may be uncertain to what extent you can trust him to give you the best advice. The salesperson wants to sell you a computer, which means that his motivations might be consciously or unconsciously influencing the information he is providing you.

Who should you trust? We have all been in situations like this, and there is no easy way to answer that question. Chances are, you’ll make your computer decision based on your interpretation of the evidence and based on what you perceive to be the reliability and credibility of your different sources. If someone else were faced with the same computer decision and the same evidence, they might make a different choice. That is why there are different kinds of computers on the market and that is why different people can do the same sort of research about “the best” computer and why they can arrive at different conclusions.

Academic research is not much different in the sense that different researchers, considering the same or similar evidence, often arrive at different conclusions. Academic research rarely provides clear answers in the sense of definitively knowing the “rights” and “wrongs” about some issue. Not all academics think that computer hacking is wrong (or right), that the solution to commercial over-fishing is strict international control, or that F. Scott Fitzgerald’s novel The Great Gatsby depicts the connection between material goods and the American dream. Rather, there are debates about these issues, differences of interpretation and opinion that result from different researchers looking at the same evidence.

Furthermore, the debates about differences of opinion on how to interpret evidence are good and healthy because these discussions further our understanding of complex issues. If we all agreed that something was true, then there would be no point in conducting research and writing about it. Indeed, if we all agreed about everything and had all of our questions answered as well as we thought possible, there would be no point to education at all!

Ultimately, there is no easy formula for evaluating the credibility and reliability of research. But there are some basic questions you should ask about your all of your evidence to ensure it is reliable and credible:

• Who wrote it?

What do you think motivated the writer?

Where was it published, when was it written.

Who wrote or said it?

Is there an author named with the evidence?

If your evidence does not name the author, it might still be reliable, especially if you have confidence about where the evidence was published. However, most credible and reliable publications tell readers who wrote the articles they contain.

On Web pages and other Internet-based sources, it can sometimes be tricky to find the name of the Web page’s author. Many web sites don’t name an author, which, given the nature of the Web, should send up red flags for you as a researcher regarding the credibility of the evidence. But like print publications, more credible Web pages will include the name of the page’s writer. Be sure to look for the writer’s name throughout the particular page (including the bottom) and related pages within the Web site.

What are the qualifications of the author?

Does he or she seem to be an expert in the field?

Have he or she written about this topic before?

Are there other experiences that seem to uniquely qualify him or her as a reliable and credible source on this topic?

Many academic publications will give a lot of detail about their authors, including their degrees and academic training, the institution where they work (if they are a college professor or instructor), and other publications they have had in the past. Popular sources tend to include less information about their writers, though they too will often indicate in a byline (where the writer’s name is listed in a magazine or newspaper article) if the writer is a reporter, contributing editor, or editor for a particular subject.

Credible web sources will also describe the qualifications of the source’s author or authors. If you can find an author’s name on a Web site but you can’t find anything about their qualifications on their research subject, you should be suspicious about what that research has to say.

Have you come across the writer based on some of the other research you have done?

After you have conducted a bit of research on your topic, you might find yourself coming across the same authors writing similar articles in different publications. You might also find different publications referring to the author or her work, which would suggest that the author is indeed reliable and credible in her field. After all, if other articles and writers refer positively to a particular writer or her articles again and again, then it seems likely that the often-referred-to writer is credible.

Understanding and trusting the expertise of the author of your evidence is probably the most crucial test of credibility and reliability of that evidence.

Simply put, academics find evidence that comes from an author who is a credible expert to be much more persuasive than evidence that does not come from an expert.

For example, while my mom is a reliable source of information regarding many different topics, it would do you little good for me to interview her for an academic research project about the problems of over-fishing. Mind you, I value my mom’s thoughts and wisdom, and she might have some things to say about the effects of decreased catches of fish that I find insightful. However, because my mom doesn’t have any expertise about commercial fishing and because she doesn’t know anything more (or less) about it than most people, most of the readers of my research project won’t be persuaded by what she has to say.

On the other hand, my mother was a hospice work for many years, working with terminally ill patients and their families. If I were conducting research about the advantages and disadvantages of hospice care for terminally ill patients, my mom might be a very interesting and credible source.

Is the writer identified with a particular organization or group that might have a specific interest in the subject of the writing?

This can often be the source of conscious or unconscious bias. An obvious example: a writer who is identified as a member of the National Riflemen’s Association, which represents a variety of Americans particularly interested in protecting the right to own guns, will certainly have a different view on gun ownership than a member of The Center to Prevent Handgun Violence, an organization working to enact gun control legislation.

You need to be particularly careful with Web-based sources of research when considering the writer’s affiliation with different groups or organizations. There have been numerous incidents where Web page writers falsely claimed their Web pages were affiliated with particular groups or causes.

Does the writer identify himself or herself with an explicit political group or party?

Considering a writer’s politics is particularly important when thinking about the credibility of a Web site. Besides the ease with which a writer can misrepresent themselves or others, the low cost and wide reach of the Web has also made it an attractive forum for hate groups, terrorists, and other “fringe” political movements. This doesn’t automatically mean the information you find on reactionary or radical Web sites is wrong; however, writers with particularly strong and extreme politics frequently present information that is biased to the point of inaccuracy.

Of course, while it is important to consider why a writer wrote about her subject and to think about how her motivations impact how she wrote about his or her subject, having a particular bias or motivation doesn’t automatically lead to a lack of credibility or reliability.

Was the piece of writing published in an academic or non-academic source? A book, a journal, a magazine, etc.? I’ve already discussed this a great deal in this chapter; generally speaking, academic sources are considered more credible than non-academic sources, and print-based sources are generally considered more credible than web-based sources.

But there are some more subtle tests of credibility and reliability concerning where a piece of research was published. For example, single-authored or co-authored scholarly books on a particular subject might be more regarded as more credible than a scholarly journal article because books go into much greater detail on topics than journal articles.

Are you familiar with the publication? If you are a new researcher to a particular field of study this can be a difficult question to answer since you might not have heard of some of the more well-known and credible publications known in that field. But once you get to know the field better (which will inevitably be the case as you conduct more research on your topic), chances are you will begin to realize certain publications are seen by experts in the field as more credible than others.

Last, but far from least, the date of publication can dramatically effect the credibility of your research. Obviously, this is especially important for date-sensitive research topics. If you were writing a research project about the Internet and the World Wide Web, chances are any research older than about 1990 or so would be of limited use since the Web literally did not exist before 1990.

But other potentially less obvious topics of research have date sensitive components to them. For example, if you were doing research on cigarette smoking or drunk driving, you would have to be careful about evaluating the credibility of research from the 1970s or 1960s or earlier since cultural “norms” in the United States for both smoking and drinking have changed a great deal.

Knowing (or rather, not knowing) the date of publication of a piece of research is yet another thing to be worried about when evaluating the credibility of Web-based sources. Many Web sites do not include any information about the date of publication or the date when the page was last updated. This means that you have no way of knowing when the information on that dateless page was published.

The date of publication is a key piece of information, the sort of thing that is always included in more print sources. Again, just because the date of publication or update is missing from a Web site does not automatically discount it as a credible source; however, it should make you suspicious.

Exercise 1.5

Working alone or collaboratively in small groups, consider a variety of different types of research—articles from scholarly and non-scholarly sources, newspaper articles, books, web sites, and other types of evidence. Using the criteria discussed here, how would you rate the quality and credibility of your research? Which of your sources seems the most reliable? Are there any pieces of evidence that, upon closer examination, do not seem credible or reliable?

Evidence Quality and Credibility Checklist

• The writer’s name
• Qualifications
• Expertise in the field
• Previous publications on the topic
• Unique experiences of the writer

Why did the source write or say it?

• Association with an organization or group
• The writer’s stated or implied politics

Where (what source) was it published?

• Academic/scholarly source versus non-academic/popular source
• Prior knowledge of publication

When was it published or said?

And when it comes to evidence from the ‘net and World Wide Web…

• It’s still important to know who wrote it, why you think they wrote it, where you found it online, and when was it published.
• If you don’t know the answers to the who/why/where/when questions, you should be skeptical of the evidence.
• Don’t be fooled by Web sites that “look” real, because…
• Anybody can publish information on the Web, no matter what that information is. Unlike most scholarly and many non-scholarly publications, Web writers don’t have to have the work reviewed by editors and publishers to reach an audience.
• The Internet and the World Wide Web are still good places to find research. You just have to be a bit more careful with them.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Working with sources
• What Are Credible Sources & How to Spot Them | Examples

What Are Credible Sources & How to Spot Them | Examples

Published on August 26, 2021 by Tegan George . Revised on May 31, 2023.

A credible source is free from bias and backed up with evidence. It is written by a trustworthy author or organization.

There are a lot of sources out there, and it can be hard to tell what’s credible and what isn’t at first glance.

Evaluating source credibility is an important information literacy skill. It ensures that you collect accurate information to back up the arguments you make and the conclusions you draw.

Table of contents

Types of sources, how to identify a credible source, the craap test, where to find credible sources, evaluating web sources, other interesting articles, frequently asked questions.

There are many different types of sources , which can be divided into three categories: primary sources , secondary sources , and tertiary sources .

Primary sources are often considered the most credible in terms of providing evidence for your argument, as they give you direct evidence of what you are researching. However, it’s up to you to ensure the information they provide is reliable and accurate.

You will likely use a combination of the three types over the course of your research process .

Prevent plagiarism. Run a free check.

There are a few criteria to look at right away when assessing a source. Together, these criteria form what is known as the CRAAP test .

• The information should be up-to-date and current.
• The source should be relevant to your research.
• The author and publication should be a trusted authority on the subject you are researching.
• The sources the author cited should be easy to find, clear, and unbiased.
• For web sources, the URL and layout should signify that it is trustworthy.

The CRAAP test is a catchy acronym that will help you evaluate the credibility of a source you are thinking about using. California State University developed it in 2004 to help students remember best practices for evaluating content.

• C urrency: Is the source up-to-date?
• R elevance: Is the source relevant to your research?
• A uthority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
• A ccuracy: Is the source supported by evidence? Are the claims cited correctly?
• P urpose: What was the motive behind publishing this source?

The criteria for evaluating each point depend on your research topic .

For example, if you are researching cutting-edge scientific technology, a source from 10 years ago will not be sufficiently current . However, if you are researching the Peloponnesian War, a source from 200 years ago would be reasonable to refer to.

Be careful when ascertaining purpose . It can be very unclear (often by design!) what a source’s motive is. For example, a journal article discussing the efficacy of a particular medication may seem credible, but if the publisher is the manufacturer of the medication, you can’t be sure that it is free from bias. As a rule of thumb, if a source is even passively trying to convince you to purchase something, it may not be credible.

Newspapers can be a great way to glean first-hand information about a historical event or situate your research topic within a broader context. However, the veracity and reliability of online news sources can vary enormously—be sure to pay careful attention to authority here.

When evaluating academic journals or books published by university presses, it’s always a good rule of thumb to ensure they are peer-reviewed and published in a reputable journal.

What is peer review?

The peer review process evaluates submissions to academic journals. A panel of reviewers in the same subject area decide whether a submission should be accepted for publication based on a set of criteria.

For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

What sources you use depend on the kind of research you are conducting.

For preliminary research and getting to know a new topic, you could use a combination of primary, secondary, and tertiary sources.

• Encyclopedias
• Websites with .edu or .org domains
• News sources with first-hand reporting
• Research-oriented magazines like ScienceMag or Nature Weekly .

As you dig deeper into your scholarly research, books and academic journals are usually your best bet.

Academic journals are often a great place to find trustworthy and credible content, and are considered one of the most reliable sources you can use in academic writing.

• Is the journal indexed in academic databases?
• Has the journal had to retract many articles?
• Are the journal’s policies on copyright and peer review easily available?
• Are there solid “About” and “ Scope ” pages detailing what sorts of articles they publish?
• Has the author of the article published other articles? A quick Google Scholar search will show you.
• Has the author been cited by other scholars? Google Scholar also has a function called “Cited By” that can show you where the author has been cited. A high number of “Cited By” results can often be a measurement of credibility.

Google Scholar is a search engine for academic sources. This is a great place to kick off your research. You can also consider using an academic database like LexisNexis or government open data to get started.

Open Educational Resources , or OERs, are materials that have been licensed for “free use” in educational settings. Legitimate OERs can be a great resource. Be sure they have a Creative Commons license allowing them to be duplicated and shared, and meet the CRAAP test criteria, especially in the authority section. The OER Commons is a public digital library that is curated by librarians, and a solid place to start.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

It can be especially challenging to verify the credibility of online sources. They often do not have single authors or publication dates, and their motivation can be more difficult to ascertain.

Websites are not subject to the peer-review and editing process that academic journals or books go through, and can be published by anyone at any time.

When evaluating the credibility of a website, look first at the URL. The domain extension can help you understand what type of website you’re dealing with.

• Educational resources end in .edu, and are generally considered the most credible in academic settings.
• Advocacy or non-profit organizations end in .org.
• Government-affiliated websites end in .gov.
• Websites with some sort of commercial aspect end in .com (or .co.uk, or another country-specific domain).

In general, check for vague terms, buzzwords, or writing that is too emotive or subjective . Beware of grandiose claims, and critically analyze anything not cited or backed up by evidence.

• How does the website look and feel? Does it look professional to you?
• Is there an “About Us” page, or a way to contact the author or organization if you need clarification on a claim they have made?
• Are there links to other sources on the page, and are they trustworthy?
• Can the information you found be verified elsewhere, even via a simple Google search?
• When was the website last updated? If it hasn’t been updated recently, it may not pass the CRAAP test.
• Does the website have a lot of advertisements or sponsored content? This could be a sign of bias.
• Is a source of funding disclosed? This could also give you insight into the author and publisher’s motivations.

Social media posts, blogs, and personal websites can be good resources for a situational analysis or grounding of your preliminary ideas, but exercise caution here. These highly personal and subjective sources are seldom reliable enough to stand on their own in your final research product.

Similarly, Wikipedia is not considered a reliable source due to the fact that it can be edited by anyone at any time. However, it can be a good starting point for general information and finding other sources.

Checklist: Is my source credible?

My source is relevant to my research topic.

My source is recent enough to contain up-to-date information on my topic.

There are no glaring grammatical or orthographic errors.

The author is an expert in their field.

The information provided is accurate to the best of my knowledge. I have checked that it is supported by evidence and/or verifiable elsewhere.

My source cites or links to other sources that appear relevant and trustworthy.

There is a way to contact the author or publisher of my source.

The purpose of my source is to educate or inform, not to sell a product or push a particular opinion.

My source is unbiased, and offers multiple perspectives fairly.

My source avoids vague or grandiose claims, and writing that is too emotive or subjective.

[For academic journals]: My source is peer-reviewed and published in a reputable and established journal.

[For web sources]: The layout of my source is professional and recently updated. Backlinks to other sources are up-to-date and not broken.

[For web sources]: My source’s URL suggests the domain is trustworthy, e.g. a .edu address.

Your sources are likely to be credible!

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

• ChatGPT vs human editor
• ChatGPT citations
• Is ChatGPT trustworthy?
• Using ChatGPT for your studies
• What is ChatGPT?
• Chicago style
• Paraphrasing

 Plagiarism

• Types of plagiarism
• Self-plagiarism
• Avoiding plagiarism
• Academic integrity
• Consequences of plagiarism
• Common knowledge

A credible source should pass the CRAAP test  and follow these guidelines:

• The information should be up to date and current.
• For a web source, the URL and layout should signify that it is trustworthy.

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

The CRAAP test is an acronym to help you evaluate the credibility of a source you are considering using. It is an important component of information literacy .

The CRAAP test has five main components:

• Currency: Is the source up to date?
• Relevance: Is the source relevant to your research?
• Authority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
• Accuracy: Is the source supported by evidence? Are the claims cited correctly?
• Purpose: What was the motive behind publishing this source?

Academic dishonesty can be intentional or unintentional, ranging from something as simple as claiming to have read something you didn’t to copying your neighbor’s answers on an exam.

You can commit academic dishonesty with the best of intentions, such as helping a friend cheat on a paper. Severe academic dishonesty can include buying a pre-written essay or the answers to a multiple-choice test, or falsifying a medical emergency to avoid taking a final exam.

To determine if a source is primary or secondary, ask yourself:

• Was the source created by someone directly involved in the events you’re studying (primary), or by another researcher (secondary)?
• Does the source provide original information (primary), or does it summarize information from other sources (secondary)?
• Are you directly analyzing the source itself (primary), or only using it for background information (secondary)?

Some types of source are nearly always primary: works of art and literature, raw statistical data, official documents and records, and personal communications (e.g. letters, interviews ). If you use one of these in your research, it is probably a primary source.

Primary sources are often considered the most credible in terms of providing evidence for your argument, as they give you direct evidence of what you are researching. However, it’s up to you to ensure the information they provide is reliable and accurate.

Always make sure to properly cite your sources to avoid plagiarism .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, May 31). What Are Credible Sources & How to Spot Them | Examples. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/working-with-sources/credible-sources/

Is this article helpful?

Tegan George

Other students also liked, applying the craap test & evaluating sources, how to cite a wikipedia article | apa, mla & chicago, primary vs. secondary sources | difference & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

University Libraries

• Research Guides
• Blackboard Learn
• Interlibrary Loan
• Study Rooms
• University of Arkansas

Credibility: Identifying Reputable Sources for Papers and Projects

• Getting Started
• Choosing Resources to Support Your Topic
• How Long Should Research Take?
• Other Factors for Choosing Reputable Resources
• How to Tell What's What

Necia Parker-Gibson

What do you need for this project? Is it a paper, a presentation, a poster?

What is your topic? What kind of a project is it? What is needed may depend on your purpose and context.

How much support do you need? Do you need scholarly materials, or will general information do? Who is writing about your topic? For what purpose(s)? This can make a difference to what you look for, and where. For most academic classes, some of your sources should be of scholarly quality-- that is, articles from peer-reviewed journals, proceedings from important conferences, and the like. The level or complexity of information required often rises with the class level-- that is, most of the time you'll need more and better sources for a senior class than for a first year class, and more if higher amounts of credit are offered for a class.

Some topics are well covered in books, or you can gain a beginning understanding via an entry in a good encyclopedia, or even Wikipedia (it keeps up well with popular culture, for example). Others are better covered in newspapers, magazines, or journal articles; videos and podcasts may be useful for some topics.

Who is considered an expert may depend on what the topic is! Being well-known (notoriety or celebrity) doesn't mean that a person is an expert. Or they may be an expert on one thing and not another; relatively few people are experts in several fields, and anyone can have an opinion.

Are you looking for scholars or practitioners, authors whose credentials are in their years of experience, a combination of those factors, or something else?

The more you know about a topic, the easier it is to evaluate what you need for a particular project- up to a point. Experience counts; having a feel for the field of interest makes a difference, too. This is one reason professors sometimes suggest topics; sometimes they suggest a topic that they want to know more about, too.

If you are giving a panel presentation, you may not need as much substantial supporting materials for yourself and your piece of the panel 'pie' as you might for a capstone project or thesis, but you will need to coordinate content with the other panelists. A simple topic may need less support than a complex one, but a complicated topic may be more interesting.

How much time do you have?

How long/how involved does the paper or presentation need to be?

What kind(s) of information do you need-- a single fact, a series of facts that build on each other? a simple analysis, an in-depth discussion, or more?

When did you start working on it? When is it due?

Some information is easier to find quickly in certain places. Specific facts such as boiling points of substances are easier to find in handbooks, or resources like Knovel. Some information is proprietary, and can't be had without paying for it; sometimes we have sources that contain it because of our paid subscriptions.

Are specific types or numbers of sources required? Articles from peer-reviewed or scholarly journals are often required; they are normally considered more reputable than magazines or some other publications, such as blogs, trade magazines, or newspapers. Many of the databases will allow you to limit your search to peer-reviewed or scholarly journals (see the box for a definition of peer review).

What Makes Authority? It's training, experience and practice.

There are exceptions, but most people who are considered authorities:

• Have studied topic X  for a long time, and in detail.
• Have practiced a particular skill, in their field, and in depth.
• Hold degrees or other credentials that represent that study
• Use sound and reputable research practices
• Perform research and/or do work that has results which can be reproduced
• Work with people who recognize their standing
• Have been cited in the literature of their field(s), and not for their mistakes
• Work at places that have good reputations in that field
• Are better known in their own field(s) than in general. Neil deGrasse Tyson may be an exception to the rule.

Peer-reviewed/Scholarly/Refereed?

Peer review is the process by which articles or other works are critiqued before they are published. Authors send articles to an editor, who decides whether the work should be forwarded to reviewers for the journal. Part of this is the content, and part of the decision is whether the manuscript matches the scope of the publication.  The most stringent form of peer review is anonymous or blind review, where neither the author nor the reviewers know whose work is being examined by whom. This helps reduce bias.

Reviewers are usually well-published researchers and experts. The reviewers return the articles to the editor with remarks and recommendations-- usually publish 'as is' (rare), publish if edited or changed in specific ways, or don't publish. Editors most often go with the recommendation of the majority of the reviewers.

The process is intended to improve the content of studies published-- more eyes on a project, and one's reputation on the line with peers, tends to improve the quality of what's submitted and published. There are cases where it hasn't worked, and critics of the cycle, but it is the best system that has been developed to this point.

Search OneSearch for Books, Journals, Videos, and more

Director for research & instruction.

• Next: Choosing Resources to Support Your Topic >>
• Last Updated: Jan 8, 2024 10:05 AM
• URL: https://uark.libguides.com/credibility
• See us on Instagram
• Follow us on Twitter
• Phone: 479-575-4104

What makes an online review credible? A systematic review of the literature and future research directions

• Open access
• Published: 05 December 2022

Cite this article

You have full access to this open access article

• K. Pooja   ORCID: orcid.org/0000-0001-7735-8308 1 &
• Pallavi Upadhyaya   ORCID: orcid.org/0000-0003-4523-2051 2  

11k Accesses

5 Citations

1 Altmetric

Explore all metrics

Online reviews of products and services are strategic tools for e-commerce platforms, as they aid in consumers’ pre-purchase decisions. Past research studies indicate online reviews impact brand image and consumer behaviour. With several instances of fake reviews and review manipulations, review credibility has become a concern for consumers and service providers. In recent years, due to growing webcare attitude among managers, the need for maintaining credible online reviews on the e-commerce platforms has gained attention. Though, there are several empirical studies on review credibility, the findings are diverse and contradicting. Therefore, in this paper, we systematically review the literature to provide a holistic view of antecedents of online review credibility. We examine variables, methods, and theoretical perspective of online review credibility research using 69 empirical research papers shortlisted through multi-stage selection process. We identify five broad groups of antecedents: source characteristics, review characteristics, consumer characteristics, interpersonal determinants in the social media platform and product type. Further, we identify research issues and propose directions for future research. This study contributes to existing knowledge in management research by providing the holistic understanding of the “online review credibility” construct and helps understand what factors lead to consumers’ belief in the credibility of online review. The insights gained would provide managers adequate cues to design effective online review systems.

Similar content being viewed by others

Online influencer marketing

Fine F. Leung, Flora F. Gu & Robert W. Palmatier

Customer engagement in social media: a framework and meta-analysis

Fernando de Oliveira Santini, Wagner Junior Ladeira, … Barry J. Babin

Artificial intelligence in E-Commerce: a bibliometric study and literature review

Ransome Epie Bawack, Samuel Fosso Wamba, … Shahriar Akter

Avoid common mistakes on your manuscript.

1 Introduction

Online reviews of products and services have become an integral component of product information on e-commerce platforms and are often used as strategic instrument to gain competitive advantage (Gutt et al. 2019 ). They are influential in marketing communications and help shoppers identify the products (Chen and Xie 2008 ) and make informed pre-purchase decisions (Hong and Pittman 2020 ; Eslami et al. 2018 ; Klaus and Changchit 2019 ; Reyes- Menendez et al. 2019 ). In the absence of physical interaction with the product, they aid consumers to take decisions based on experiences shared by previous users on the e-commerce platform (Klaus and Changchit 2019 ). Reviews facilitate the free flow of consumer-generated content that help managers promote their products or brand or company (Smith 2011 ). The products that get at least 5 reviews have a 270% higher conversion rate compared to the products with no reviews (Collinger et al. 2017 ).

With the growing popularity of online reviews, there is an overwhelming interest among researchers to understand the characteristics of reviews and reviewer that contribute to the credibility of online reviews (Cheung et al. 2009 ; Chih et al. 2020 ; Fang and Li 2016 ; Jimenez and Mendoza 2013 ; Liu and Ji 2018 ; Mumuni et al. 2019 ; Qiu et al. 2012 ; Tran and Can 2020 ; Yan et al. 2016 ). The credibility of online information and digital media is often contested, due to the lack of quality control standards and ambiguity concerning the ownership of the information with the convergence of information and media channels (Flanagin and Metzger 2007 ). As all online reviews cannot be trusted (Johnson and Kaye 2016 ) and when sources are uncertain (Lim and Van Der Heide 2015 ) consumers often use cues to assess review credibility. The credibility issue also arises due to review manipulation practices by asking the reviewers to write a positive review in favour of the brand and to write a negative review attacking the competitor's product, by incentivizing the reviewer (Wu et al. 2015 ).

Recent meta-analysis studies on electronic word of mouth (eWOM) communications have focused on factors impacting eWOM providing behaviour (Ismagilova et al. 2020a ), the effect of eWOM on intention to buy (Ismagilova et al. 2020b ), the effect of source credibility on consumer behaviour (Ismagilova et al. 2020c ), factors affecting adoption of eWOM message (Qahri-Saremi and Montazemi 2019 ) and eWOM elasticity (You et al. 2015 ). Moran and Muzellec ( 2017 ) and recently Verma and Dewani ( 2020 ) have proposed four-factor frameworks for eWOM Credibility. Zheng ( 2021 ) presented a systematic review of literature on the classification of online consumer reviews.

Even though there are literature reviews and meta-analysis on eWOM, they address different research questions or constructs in eWOM and no attempt to synthesise the antecedents of online review credibility, in the context of products and services has been made. Xia et al. ( 2009 ) posit that all eWOM are not formulated equally and classify eWOM as “many to one” (e.g., No of ratings, downloads calculated by computers), “many to many” (e.g., Discussion forums), “one to many” (e.g., Text-based product reviews), and “one to one” (instant messaging). Studies confirm that the effort to process and persuasiveness of different forms of eWOM vary (Weisfeld -Spolter et al. 2014 ). Senecal and Nantel ( 2004 ) argue that consumers spend significantly more time and effort to process online reviews than any other form of eWOM. Hence understanding credibility of the online reviews and the factors that influence credibility is important for managers of e-commerce platforms.

Our objective in this paper is three-fold: First, we revisit, review, and synthesize 69 empirical research on online review credibility that focuses on textual online reviews of products and services (“one to many” form of eWOM). Second, we identify the antecedents of review credibility. Finally, we identify gaps and propose future research directions in the area of online reviews and online review credibility. From theoretical perspective, this systematic review synthesises the antecedents of review credibility, in the context of online reviews of products and services. As in past literature, eWOM and online reviews are interchangeably used, we carefully analysed both the eWOM credibility and online review credibility and selected studies that focused on reviews of products and services. Studies on sponsored posts on social media, blogs, the brand initiated eWOM communication were excluded. From managerial perspective, this study would aid managers of e-commerce platforms, a holistic view of review credibility and aid in the design of online review systems.

1.1 Defining online review credibility

Mudambi and Schuff ( 2010 ) define online reviews as “peer-generated product evaluations, posted on company or third-party websites”. Person-to-person communication via the internet is eWOM. An online review is a form of eWOM. There are various channels of eWOM such as social media, opinion forums, review platforms, and blogs. Past literature posits that credible eWOM is one that is perceived as believable, true, or factual (Fogg et al. 2001 ; Tseng and Fogg. 1999 ).

The perception a consumer holds regarding the veracity of online review is considered as the review credibility (Erkan and Evans 2016 ). Several research studies (Cheung et al. 2009 ; Dong 2015 ) define credible online reviews as a review that the consumers perceive as truthful, logical, and believable. Past research defines credibility to be associated with consumers’ perception and evaluation and not as a direct measure of the reality of reviews (Chakraborty and Bhat 2018a ). The credibility of online reviews is described as consumers’ assessment of the accuracy (Zha et al. 2015 ) and validity of the reviews (Chakraborty and Bhat 2017 ).

2 Research methods

This paper uses the systematic literature review method (Linnenluecke et al. 2020 ; Moher et al. 2009 ; Neumann 2021 ; Okoli 2015 ; Snyder 2019 ) to synthesize the research findings. Liberati et al. ( 2009 ) explains systematic review as a process for identifying, critically appraising relevant research and analyzing data. Systematic reviews differ from meta-analysis with respect to methods of analysis used. While meta-analysis focuses primarily on quantitative and statistical analysis; systematic reviews use both quantitative and qualitative analysis and critical appraisal of the literature. In a systematic review, pre-specified protocols on inclusion and exclusion of the articles are used to identify the evidence that fits the criteria to answer the research question (Snyder 2019 ). In this paper, we follow the steps proposed by Okoli ( 2015 ) for conducting the systematic review process and the recommendations given by Fisch and Block ( 2018 ) to improve the quality of the review. The purpose of our systematic literature review is to identify and synthesize the antecedents of online review credibility.

The study uses journal articles from two popular research databases (Scopus and Web of Science) to conduct a systematic search of articles on review credibility/eWOM credibility. As online reviews are interchangeably used with other related concepts such as eWOM, user-generated content, and online recommendations in the literature, we used a diverse pool of sixteen keywords (refer Fig.  1 ) for the initial search. The keywords were identified through an initial review of literature and articles having these terms in the title, abstract, and keywords were chosen. Initial search and document retrieval were done in January 2022. Studies published till October 2022 were later updated in the paper. A set of filters using inclusion and exclusion criteria were applied to arrive at a focused set of relevant papers. The full-length empirical articles in English language, related to business management and allied areas were included for systematic review. Using multiple phases of filtering and reviewing (refer Fig.  1 ), we shortlisted the final list of 69 empirical papers that used either review credibility or eWOM credibility as a construct with a focus on reviews of products and services. In line with previous systematic reviews (Kuckertz and Brändle 2022 ; Nadkarni and Prügl 2021 ; Walter 2020) we excluded work in progress papers, conference papers, dissertations or books from the analysis.

Systematic review process

2.1 Descriptive analysis of empirical research on online review credibility

The 69 empirical research articles included 36 experimental design studies and 33 cross-sectional survey-based studies. Figure  2 summarises the review credibility publication trends in the last decade with their research design choices.

Research designs of Review credibility articles

Research on review credibility has used samples from diverse geographical regions, the highest number of studies being in the USA, China, and Taiwan (refer to Table 1 ). Table 2 and Table 3 summarizes the sample and analysis methods used in these studies. Even though online review is commonly used in tourism and hospitality, there are only six studies examining review credibility.

3 Theoretical perspectives in review credibility literature

Most of the empirical research (88 percent) on review credibility has used theories to explain the antecedents of review credibility. A total of 48 different theories have been invoked in explaining various dimensions of review credibility antecedents.

We observed five broad groups of theories from the underlying 48 theories that contribute to understanding the different aspects of online review credibility assessment by consumers. We discuss them in the following sections.

3.1 Information processing in online review

Several theories provide a lens to understand ways in which individual consumes or processes the information available in the online reviews. The popular theories discussed in the review credibility literature such as the elaboration likelihood model, heuristic—systematic model, accessibility—diagnosticity theory, and attribution theory describe how an individual processes information.

Building on the elaboration likelihood model (ELM) several studies have examined characteristics of online review content such as argument quality (Cheung et al. 2009 ; Hussain et al. 2018 ; Thomas et al. 2019 ), review sidedness (Cheung et al. 2012 ; Brand and Reith 2022 ), review consistency (Brand et al. 2022 ; Brand and Reith 2022 ; Cheung et al. 2012 ; Thomas et al. 2019 ), and source credibility (Cheung et al. 2012 ; Hussain et al. 2018 ; Reyes- Menendez et al. 2019 ). These dimensions are also examined using the heuristics-systematic model (HSM). These two theories are similar in their function as both ELM and HSM posit two routes (the central vs. peripheral route and the systematic vs. heuristic route) for judging the persuasiveness of messages (Chang and Wu 2014 ). In literature, the elaboration likelihood model has received more empirical support compared to the heuristics systematic model. The yale persuasive communication theory covers a wider array of factors that can affect the acceptance of the message (Chang and Wu 2014 ). This theory has been adopted by studies to evaluate the relationship between these factors with review credibility.

The psychological choice model posits that the effectiveness of online reviews gets influenced by environmental factors like product characteristics and consumer’s past experience. These factors influences the credibility assessment by the consumer and purchase decision based on their interaction with the online reviews.

Consumers’ use of information for judgment also depends upon the accessibility and diagnosticity of the input as proposed in accessibility-diagnosticity theory. This theory helps in understanding the utilization of information by individuals and posits that the information in hand has more value than information stored as a form of memory (Tsao and Hseih 2015 ; Chiou et al. 2018 ). The attribution theory helps in understanding the nature of the causal conclusion drawn by the consumers in the presence of negative and positive information (Chiou et al. 2018 ).

Overall, the theories related to information processing have contributed well to understanding the influence of strength of the message, argument, valence, source reputation, consistency, persuasiveness, and diagnosability.

Theories such as media richness theory (Tran and Can 2020 ) and language expectancy theory (Seghers et al. 2021 ) provided insights into the relevance of the quality of the information shared in online reviews. Several other theories focus on the information adoption process (ex. Information adoption mode, informational influence theory, dual-process theory). For example, cognitive cost theory has been used to explain review adoption due to the effect of different levels of cognitive involvement of the consumer when they are exposed to reviews from different platforms simultaneously (Yan et al. 2016 ).

The contribution of technology acceptance model (TAM) to the review credibility literature is operationalized in the study by Liu and Ji ( 2018 ). Hussain et al. ( 2018 ) uses TAM to complement ELM in the computer-mediated communication adoption process.

We observe that the theories in information processing in the online review have provided a theoretical lens to understand the role of the quality of the information in the online review credibility assessment.

3.2 Trust in online reviews

Studies have examined the trust formation and perception of the trustworthiness of the source of the information in online reviews using the theoretical lens of trust transfer theory and source credibility theory. Virtual communities do not support the face-to-face interaction between sender and receiver of the message. Therefore, the receiver has to rely on cues such as the reputation of the source, credibility of the source, and the reviewer profile. These cues are observed as some of the antecedents of review credibility. Trust transfer theory contributes to our understanding of how online reviews shared on a trusted e-commerce website makes the consumer consider that review is credible compared to the review shared on a website that is not trustworthy (Park and Lee 2011 ). Source credibility theory suggests trustworthiness and expertise of the source of the review have a positive relationship with review credibility (Mumuni et al. 2019 ; Shamhuyenhanzva et al. 2016 ). These theories note that when a person perceives the origin of online review as trustworthy, he would be more likely to consume the information.

3.3 Socio-cultural influence in online reviews

Individuals’ innate values or beliefs help shape their behaviour. As online reviews are more complex social conversations (Kozinets 2016 ) there is a need to gain perspectives on how these conversations differ in terms of country and culture (Bughin et al. 2010 ). The theories such as culture theory, and Hall’s categorization provide a lens to examine the influence of culture on online review consumption and assessment of review credibility (Brand and Reith 2022 ; Chiou et al. 2014 ; Luo et al. 2014 ).

In general, attention paid to understanding the influence of cultural factors on online reviews is very limited (Mariani et al. 2019 ; Gao et al. 2017 ). However, much attention has been given to understanding the role of social influence through the use of theories like social influence theory, role theory, social identity theory, social information processing theory, socio-cognitive systems theory, and value theory. The most prominent theory related to this theme is the social influence theory. Social influence theory emphasizes the social pressure faced by consumers to form a decision based on online reviews (Jha and Shah 2021 ). Social identity theory posits that an individual may reduce uncertainty by choosing to communicate with other people who share similar values and social identities (Kusumasondjaja et al. 2012 ).

Social information processing theory posits the importance of the closeness between review writer and reader on social networking as an alternative cue, in the absence of physical interaction (Lim and Van Der Heide 2015 ). The social standings of an individual in terms of the number of friends on social networks (Lim and Van Der Heide 2015 ), nonverbal cues such as profile photos (Xu 2014 ), and their impact on review credibility have been studied using this theory. In a nutshell, these theories explain individuals’ belief that gets shaped due to the influence of the social groups and how it impacts the credibility of the review.

3.4 Consumer attitude and behaviour towards online reviews

Consumers attitude towards computer-mediated communications and online reviews have been examined in past studies (Chakraborty and bhat 2017 ; Chih et al. 2020 ; Hussain et al. 2018 ; Isci and Kitapci 2020 ; Jha and Shah 2021 ) using several theoretical frameworks. Theories such as attitude—behaviour linkage, cognition-affection-behaviour (CAB) model, expectancy-disconfirmation theory (EDT), needs theory, regulatory focus theory, search and alignment theory, stimulus- organism-response model, theory of planned behaviour, yale attitude change model, associative learning theory were used in literature to examine the factors that influence the formation of the attitude and behaviour towards online reviews. These factors and their relationship with credibility evaluation have been studied by the yale attitude change model (Chakraborty and Bhat 2017 , 2018b ), and the stimulus-organism-response model (Chakraborty 2019 ). Jha and Shah ( 2021 ) adapted attitude-behavior linkage theory to study how the exposure to past reviews acts as an influence to write credible reviews.

The consumer’s expectation about product experience and credibility assessment is studied using theories like expectancy-disconfirmation theory (Jha and Shah 2021 ), needs theory (Anastasiei et al. 2021 ), and regulatory focus theory (Isci and Kitapci, 2020 ; Lee and Koo, 2012 ). Overall, these theories have contributed to the advancement of the understanding of the holistic process involved in consumer attitude formation and behaviour in online reviews.

3.5 Risk aversion

The theories such as category diagnosticity theory, prospect theory, uncertainty management theory, and uncertainty reduction theory provide a theoretical lens to examine how consumers rely on credible information to avoid uncertain outcomes. Hong and Pittman ( 2020 ) use category diagnosticity theory and prospect theory to hypothesize negative online reviews as more credible than positive reviews. An individual who focuses on reducing loss perceives negative online reviews as more diagnostic and credible. Kusumasondjaja et al. ( 2012 ) also argue that consumers try to avoid future losses by spending effort to find credible information before making a decision. With the help of these underlying assumptions, studies have used perspectives drawn from theories to understand the loss-aversion behaviour and higher perceived diagnostic value of negative information. Prospect theory suggests consumers attempt to avoid risks or loss and expect gain. Consumers avoid choosing the experience which has more negative online reviews because of the risk and loss associated with the negativity of the reviews (Floh et al. 2013 ). The risk aversion-related theories have contributed to understanding the consumers’ quest for credible information in negative reviews.

4 Antecedents of online review credibility

Literature on review credibility reveals varied nomenclature and operationalisation of antecedents of review credibility. However, we can broadly categorize review credibility antecedents into five broad groups: source characteristics, message characteristics, consumer characteristics, social/interpersonal influence, and product type (Refer to Fig.  3 ).

Anteeedents of review credibility

We discuss these antecedent themes along with the major constructs in each theme in the following sections. In the final section, we also summarise the theoretical perspectives in each antecedent themes.

4.1 Source characteristics

Literature reveals that several characteristics of the source influence the credibility perception and evaluation of review by consumers. Chakraborty and Bhat ( 2017 ) define a source as the person who writes online reviews. Researchers have operationalized the source characteristics primarily through reviewers’ knowledge and reliability (Chakraborty and Bhat 2017 ); reviewer characteristics such as identity disclosure, level of expertise, review experience, and total useful votes (Liu and Ji 2018 ). In several studies (Cheung et al. 2012 ; Chih et al. 2013 ; Mumuni et al. 2019 ; Newell and Goldsmith 2001 ; Reyes- Menendez et al. 2019 ; Yan et al. 2016 ), expertise and trustworthiness of the reviewer is one of the most common conceptualizations of source credibility. Cheung and Thadani ( 2012 ) define source credibility as the “message source’s perceived ability (expertise) or motivation to provide accurate and truthful (trustworthiness) information”.

Source credibility is used as a single construct in several studies (Abedin et al. 2021 ; Chih et al. 2013 ; Cheung et al. 2009 , 2012 ; Mumuni et al. 2019 ; Reyes-Menendez et al. 2019 ; Yan et al. 2016 ; Luo et al. 2014 ). Studies have also conceptualized its sub-dimensions such as source trustworthiness (Chih et al. 2020 ; Lo and Yao 2018 ; Shamhuyenhanzva et al. 2016 ; Siddiqui et al. 2021 ; Thomas et al. 2019 ; Tien et al. 2018 ); reviewer expertise (Anastasiei et al. 2021 ; Fang 2014 ; Fang and Li 2016 ; Jha and Shah 2021 ) and reviewers’ authority (Shamhuyenhanzva et al. 2016 ), as separate antecedents to review credibility. Mumuni et al. ( 2019 ) posited that reviewer expertise and reviewer trustworthiness as two distinct constructs. Chih et al. ( 2020 ) define source trustworthiness as the credibility of the information presented by the message sender. Thomas et al. ( 2019 ) operationalize reviewer expertise as a peripheral cue and found that the amount of knowledge that a reviewer has about a product or service is influential in consumer’s perception of review credibility. Information presented by professional commentators who are perceived as experts in the specific field was found to have a positive influence on credibility (Chiou et al. 2014 ).

Source cues help in assessing the credibility and usefulness of the information shared in product reviews (Liu and Ji 2018 ). Reviews written by the source whose identity is disclosed have higher credibility compared to the reviews written by unidentified sources (Kusumasondjaja et al. 2012 ). However, in case of positive reviews with disclosed identity of the sponsor the review, credibility is negatively affected (Wang et al. 2022 ). Zhang et al. ( 2020 ) found that suspicion about the identity of the message sender influences negatively on the message’s credibility. Past studies found that when the number of friends of a reviewer (Lim and Van Der Heide 2015 ) and a number of trusted members of the reviewer (Xu 2014 ) are high in the online review community, reviews of such reviewers are considered as more credible. If a reviewer involves very actively in writing the review, the number of reviews posted by the reviewer provides evidence to the reader that the reviews written by such reviewers are credible (Lim and Van Der Heide 2015 ). The consumer also believes online reviews to be credible when they perceive the reviewer as honest (Yan et al. 2021) and caring (Yan et al. 2021). The source characteristics as antecedents of review credibility are summarized in Table 4 .

Several studies also define the source with the characteristics of the platform where the review is published. Consumers’ trust on the website (Lee et al. 2011 ) and the reputation of the website (Chih et al. 2013 ) were found as antecedents of the review credibility. If a consumer perceives an online shopping mall as trustworthy, he would believe that reviews posted in shopping mall as credible (Lee et al. 2011 ). Chih et al. ( 2013 ) posit that in addition to the source credibility (reviewer expertise), consumers evaluate the quality of contents of a website based on website reputation, which in turn leads to higher trust on the website and higher perceived credibility of the review. Website reputation is defined as the extent to which consumers perceive the platform where the review is published to be believable and trustworthy (Chih et al. 2013 ; Thomas et al. 2019 ; Tran and Can 2020 ; Guzzo et al. 2022 ; Majali et al. 2022 ). Bae and Lee ( 2011 ) found that consumer-developed sites were perceived as more credible than marketer-developed sites. Similarly, Tsao and Hsieh ( 2015 ) found that review quality as perceived by consumers had a higher impact on review credibility on independent platforms than on corporate-run platforms. Ha and Lee ( 2018 ) found that for credence service (eg. Hospital), the provider-driven platform and reviews were more credible and for experience goods (eg. Restaurant), consumer-driven platforms were perceived as more credible.

4.2 Review characteristics

Several characteristics of the message or the review are found to influence the review credibility on online review platforms (presented in Table 5 ). A product with a large number of reviews provides evidence of higher sales and popularity of the product (Flanagin and Metzger 2013 ; Hong and Pittman 2020 ; Reyes- Menendez et al. 2019 ). When online review for a product or service is higher, it directly influences the review credibility (Hong and Pittman 2020 ; Reyes- Menendez et al. 2019 ; Thomas et al. 2019 ; Tran and Can 2020 ).

If the reviewer agrees with most of online reviews or recommendations of others those reviews are considered as consistent reviews (Chakraborty and Bhat 2017 , 2018b ; Chakraborty 2019 ). The consistent online reviews were found to have higher credibility (Abedin et al. 2021 ; Baharuddin and Yaacob 2020 ; Brand and Reith 2022 ; Chakraborty and Bhat 2017 , 2018b ; Chakraborty 2019 ; Cheung et al. 2009 , 2012 ; Luo et al. 2014 ; Tran and Can 2020 ). Fang and Li ( 2016 ) found out that receiver of the information actively monitors the consistency of the information while perceiving the credibility of review. The degree of agreement in aggregated review ratings on the review platform creates consensus among the reviewers (Qiu et al. 2012 ). Information evolved from such consensus is perceived as highly credible (Lo and Yao 2018 ; Qiu et al. 2012 ). However, a few studies (Cheung et al. 2012 ; Luo et al. 2015 ; Thomas et al. 2019 ) have reported contradicting findings and argue that when the involvement of consumers is low and consumers are knowledgeable, review consistency has an insignificant impact on the review credibility.

Past studies have found strong evidence on the impact of review argument quality (Anastasiei et al. 2021 ; Baharuddin and Yaacob 2020 ; Cheung et al. 2012 ; Thomas et al. 2019 ; Tran and Can 2020 ; Tsao and Hsieh 2015 ) and review quality (Bambauer-Sachse and Mangold 2010 ; Chakraborty and Bhat 2017 , 2018b ; Chakraborty 2019 ; Liu and Ji 2018 ) and argument strength (Cheung et al. 2009 ; Fang 2014 ; Fang and Li 2016 ; Luo et al. 2015 ) on review credibility. Concreteness in the argument also positively impacts the review credibility (Shukla and Mishra 2021 ).

According to Petty et al. ( 1983 ), the strength of the argument provided in the message represents the quality of the message. Cheung et al. ( 2009 ) define argument strength as the quality of the information in the online review. Chakraborty and Bhat ( 2017 ) present review quality as the logical and reliable argument in the online review. Recent studies (Thomas et al. 2019 ; Tran and Can 2020 ) considered accuracy and completeness as dimensions of argument quality.

Review attribute helps in classifying the review as an objective review or subjective review based on the information captured (Lee and Koo 2012 ). Jimenez and Mendoza (2013); Gvili and Levy ( 2016 ) operationalize the level of detail as the amount of information present in the review about a product or service. Past studies have found evidence for the positive relationship between different attributes of reviews such as review objectivity (Luo et al. 2015 ; Abedin et al. 2021 ), level of detail (Jimenez and Mendoza 2013 ), review attribute (Lee and Koo 2012 ), message readability (Guzzo et al. 2022 ), persuasiveness of eWOM messages (Tien et al. 2018 ), interestingness (Shamuyenhanzva et al. 2016 ), graphics (Fang and Li 2016 ) and suspicion of truthfulness (Zhang et al. 2020 ) with review credibility. Vendemia ( 2017 ) found that the emotional content of information in the review also influences the review credibility. While assessing the review credibility, the utilitarian function of the review (Ran et al. 2021 ) and message content (Siddiqui et al. 2021 ) play an important role.

Several studies confirm that review valence influences review credibility (Lee and Koo 2012 ; Hong and Pittman 2020 ; Lo and Yao 2018 ; Manganari and Dimara 2017 ; Pentina et al. 2018 ; Pentina et al. 2017 ; vanLohuizen and Trujillo-Barrera 2019 ; Kusumasondjaja et al. 2012 ; Lim and Van Der Heide 2015 ; Chiou et al. 2018 ). Chiou et al. ( 2018 ) explain review valence is negative or positive evaluation of the product or service in online reviews. Review valence is often operationalized in experimental research at two levels: positive reviews vs negative reviews. Several studies report that negative reviews are perceived to be more credible than positive reviews (Chiou et al. 2018 ; Kusumasondjaja et al. 2012 ; Lee and Koo 2012 ; Lo and Yao 2018 ; Manganari and Dimara 2017 ). Negative reviews present a consumer’s bad experience, service failure or low quality and they create a loss-framed argument. Tversky and Kahneman ( 1991 ) explain that loss-framed arguments have a greater impact on the behaviour of consumer than gain-framed arguments. Contradictory to these findings, a few studies found that positive reviews are more credible than negative reviews (Hong and Pittman 2020 ; Pentina et al. 2017 , 2018 ). Lim and Van Der Heide ( 2015 ) found that though negative reviews impact greatly on consumer behavior it is perceived to be less credible.

Several studies (Chakraborty 2019 ; Cheung et al. 2012 ; Luo et al. 2015 ) have observed the impact of review sidedness (positive, negative or two-sided reviews) on review credibility and found that two-sided reviews are perceived as more credible. Further, Cheung et al. ( 2012 ) found that when consumers’ expertise level was high and involvement level was low, review sidedness had a stronger impact on review credibility.

Star ratings are numerical evidence of product performance (Hong and Pittman 2020 ). Star rating represents the average rating of all the review ratings therefore it helps to assess the conclusions in general (Tran and Can 2020 ). Rating evaluation needs a low amount of cognitive effort while processing the review information (Thomas et al. 2019 ). Past studies have found star ratings (Hong and Pittman 2020 ), aggregated review scores (Camilleri 2017 ), product or service ratings (Thomas et al. 2019 ; Tran and Can 2020 ), review ratings (Luo et al. 2015 ), and recommendation or information rating (Cheung et al. 2009 ) act as peripheral cues influencing the review credibility.

4.3 Consumer characteristics

Receiver is the consumer of the review and consumer needs, traits, motivation, knowledge, and involvement have been found to influence the review credibility. Chih et al. ( 2013 ) posit that online community members have two types of needs: functional need (need to find useful product information) and social need (need to build social relationships with others). These needs motivate consumers to use online reviews and form perceptions of review credibility. Consumers refer to online reviews to understand the product's pros, cons, and costs (Hussain et al. 2018 ); reduce purchase risk, and information search time (Schiffman and Kanuk 2000 ).

Past research studies indicate consumer’s motivation to obtain more information on purchase context (Chih et al. 2013 ), self-worth reinforcement (Hussain et al. 2018 ), opinion seeking from other consumers (Hussain et al. 2018 ), and prior knowledge of the receiver on the product (Cheung and Thadani 2012 ; Wang et al. 2013 ), influences review credibility. When the online reviews are congruous to the consumer’s knowledge and experiences, the message is perceived to be credible (Chakraborty and Bhat 2017 , 2018b ; Chakraborty 2019 ; Cheung et al. 2009 ). Chiou et al. ( 2018 ) found that high-knowledge consumers find reviews less credible. Studies in the past have also used prior knowledge of consumers as a control variable (Bae and Lee 2011 ) and moderating variable (Doh and Hwang 2009 ) when studying other factors. Bambauer-Sachse and Mangold ( 2010 ) found that knowledge on manipulations on product reviews influenced consumers' product evaluations, negative reviews, in particular, and when they come from a highly credible source.

Lim and Van Der Heide ( 2015 ) observed differences in the perceived credibility of users and non-users of the review platform and found an interaction effect between users’ familiarity with the review platform and reviewer profile (number of friends and number of reviews) characteristics of review credibility. Consumer experience with online reviews affects their perception of review credibility (Guzzo et al 2022 ). Izogo et al ( 2022 ) posit that consumer experiences such as sensory, cognitive and behavioral experience also influences review credibility. Consumer motivation, beliefs, and knowledge, as antecedents in literature, are summarised in Table 6 .

Cheung et. al ( 2012 ) posited that the influence of source and message characteristics on review credibility depends on two characteristics of the consumer: involvement and expertise. The authors found that level of involvement and knowledge of consumers moderate the relationships between review characteristics (review consistency and review sidedness) source credibility, and review credibility. Consumers process the information through central route, when making high involvement decisions and carefully read the content (Lin et al. 2013 ; Park and Lee 2008 ). When consumers have low involvement decisions, they are more likely to use peripheral cues and pay lesser attention to the review content, resulting in low eWOM credibility. Xue and Zhou ( 2010 ) found that consumers with high involvement decisions trusted negative reviews. In a recent study, Zhang et al. ( 2020 ) found that personality traits such as dispositional trust can trigger suspicion about the truthfulness of the message and may in turn, impact review credibility.

4.4 Interpersonal influence in the social media

Earlier research shows that interpersonal influence (Chu and Kim 2011 ) and tie strength (Bansal and Voyer 2000 ) positively influences online reviews. Consumers perceive online reviews as more credible when social status and cognitive dissonance reduction can be achieved through online forums (Chih et al. 2013 ). The previous studies have considered these factors under the theme related to source or communicator of the message (Verma and Dewani 2020 )). However, the constructs tie strength and homophily represent an interpersonal relationship between the communicator and the reader. Therefore, we discuss them separately. Tie strength is considered to be higher in an online community when the members have close relationships with other members and frequently communicate with each other. Consumers who have similar tastes and preferences share information in brand communities and enjoy meeting other members in a meaningful way (Xiang et al. 2017 ). Reviews are found to be more credible when review writers get exposed to past reviews written by others (Jha and Shah 2021 ). The exposure to past reviews moderates the relationship between disconfirmation and perception of online review credibility (Jha and Shah 2021 ). The recommendations of the members on social networking sites have also been found to be influencing the credibility of online reviews (Siddiqui et al. 2021 ).

Consumers’ perceptions of their similarity to the source of message are believed to impact their credibility assessment (Gilly et al. 1998 ; Wangenheim and Bayon 2004). Brown and Reingen ( 1987 ) define similarity or homophily as the “degree to which individuals are similar to sources in terms of certain attributes”. Herrero and Martin ( 2015 ) found that hotel consumers would perceive reviews more credible when there is a similarity between users and content creators. Source homophily is found to have an impact on review credibility in the e-commerce context as well (Abedin et al. 2021 ). Similarity of the source is often described in terms of interests of consumers and content generators. Xu ( 2014 ) posits that when a greater number of trusted members for reviewers are present on the website, it increases trust, thereby impacting the perceived credibility of the review. (Table 7 ).

4.5 Product type

The type of the product (search or experience product) is found to impact user’s evaluation of review credibility (Bae and Lee 2011 ; Jimenez and Mendoza 2013 ) and review helpfulness (Mudambi and Schuff 2010 ). Experience products differ from search products. They require more effort in retrieving product’s attribute-related information online and often require direct experience to assess the product features accurately. Bae and Lee ( 2011 ) found that when review originates from the consumer-owned online community, consumers find review credible for experience products. Tsao and Hsieh ( 2015 ) found that the credibility of eWOM is stronger for credence products than search products. Credence goods are those whose qualities cannot be confirmed even after purchase, such as antivirus software and sellers often cheat consumers due to information asymmetry and charge higher prices for inferior goods.

Jimenez and Mendoza ( 2013 ) found differences in consumers’ evaluation of review credibility for search and experience products. The study found that for search products detailed reviews were considered more credible and for experience products, reviewer agreement impacted review credibility (Jimenez and Mendoza 2013 ). Chiou et al. ( 2014 ) found that the review credibility was perceived differently for elite (eg: Classical musical concerts) and mass (eg: movies) cultural offerings. The study posited that when consumers read reviews of elite cultural offerings, and it originates from professionals, it is perceived as more credible. (Table 8 ).

4.6 Summary of antecedent themes and theoretical perspectives

Review characteristics, followed by source characteristics, are the most researched themes in terms of the number of studies and theories used (refer to Fig.  4 ). It indicates the wide coverage of different theoretical perspectives examined in these two areas. Consumer characteristics, interpersonal determinants in social media, and product type were less researched antecedent themes and lesser examined through a theoretical lens.

Anteeedent themewise articles and theories

The most popular theories in review credibility literature are the elaboration likelihood model, social influence theory, accessibility- diagnosticity theory, attribution theory, and theory of reasoned action. Contribution from these theories was noted in at least four antecedent themes identified in our study. Table 9 summarizes the theories used in each antecedent theme identified in the current review.

5 Review credibility: future research directions

Though there is ample research on online review credibility, there are several gaps in understanding the aspects of consumer behavior in online review evaluation and mitigation of issues with credibility. We identify six research issues that need further investigation and empirical evidence.

5.1 Research issue 1: review credibility in a high-involvement decision-making context

Several studies have examined credibility of reviews in experience products such as movies (Chiou et al. 2014 ; Flanagin and Metzer 2013 ), restaurants (Ha and Lee 2018 ; Pentina et al. 2017 ; vanLohuizen and Trujillo-Barrera 2019 ), hotels (Lo and Yao 2018 ; Manganari and Dimara 2017 ), and search goods such as audiobooks (Camilleri 2017 ), consumer electronics (Bambauer-Sachse and Mangold 2010 ; Chiou et al. 2018 ; Lee et al. 2011 ; Lee and Koo 2012 ; Tsao and Hsieh 2015 ; Xu 2014 ), few studies (Jimenez and Mendoza 2013 ; Doh and Hwang 2009 ; Xue and Zhou 2010 ; Bae and Lee 2011 ) have examined both experience and search products.

However, most of the products involve low to medium involvement of consumers and there is a gap in understanding online review usage, credibility, and impact in the context of high involvement decisions. There are several online review platforms on high involvement goods and services such as cars (eg: carwale, auto-drive), and destination holiday planning (TripAdvisor). Consumers often use online reviews to reduce purchase risk. As purchase risks are higher in high involvement decisions, consumers would spend more time searching online to evaluate the product. It is also necessary to understand to what extent consumers trust online reviews in a high involvement decision context, which often combines online information, reviews, and offline experiences (eg: visit to a car dealership for a test drive). Previous studies on consumer involvement (Hussain et al. 2018 ; Lin et al. 2013 ; Park and Lee 2008 ; Reyes-Menendez et al. 2019 ; Xue and Zhou, 2010 ) have operationalized involvement as a multi-item construct that captures the level of involvement of consumers, using consumers’ response. Experimental design studies, using high involvement goods and their reviews would help to establish causal relationships, in high involvement goods context. As an exception, one of the recent studies by Isci and Kitapci ( 2020 ) uses experimental design using automobile products as the stimuli for the experiment. However, as observed in our analysis, there are scarce studies in high involvement decision making context.

5.2 Research issue 2: mitigation of low credibility of the online review

While extant literature is available on factors affecting review credibility and its impact on brand and consumer behavior, there is limited literature and discussion on how companies can mitigate the impact of low credibility of reviews and improve trust. More evidence and empirical research is required to demonstrate effectiveness of measures that firms can take to build credibility and improve trust. As reviews are an important component of product information in e-commerce websites and reviews are used to form pre-purchase decisions, research on mitigation of poor credibility would be useful. For example, while past research shows that reviews on marketer-developed sites are perceived less credible for experience products than consumer-developed sites (Bae and Lee 2011 ). There is a need to study strategies that marketers can use to gain the trust of consumers.

5.3 Research issue 3: mitigating impact of negative online reviews

Past studies have indicated that consumers pay more attention to negative reviews (Kusumasondjaja et al. 2012 ; Lee and Koo 2012 ; vanLohuizen and Barrera 2019 ; Yang and Mai 2010 ), and trust (Xue and Zhou 2010 ; Banerjee and Chua 2019 ) more than positive reviews. Negative reviews are found to be persuasive and have a higher impact on brand interest and purchase intention (Xue and Zhou 2010 ). There are also limited studies discussing the ways to mitigate the impact of negative reviews and strategies to deal with them in a wide variety of contexts. While extant literature is available on review characteristics such as review sidedness, review valence, and its impact on review credibility (Refer to Table 5 ), there is little empirical evidence on strategies to deal with negative reviews. An exception is a study by Pee ( 2016 ), that addressed this issue by focusing on marketing mix and suggested that managing the marketing mix can mitigate the impact of negative reviews. However, more research is needed to equip marketers with mitigation techniques and fair strategies to deal with negative reviews.

5.4 Research issue 4: credibility of brand initiated online reviews

Brand-initiated eWOM often incentivizes consumers to share the content with their friends and it is unclear whether such initiatives are perceived as less credible. Brands use a variety of strategies to promote products on social media and facilitate person-to-person communications of brand content such as referral rewards, coupons, and bonus points (Abu-El-Rub et al. 2017 ). Incentivized reviews can easily manipulate consumers as their motive is not to provide unbiased information to make an informed decision (Mayzlin et al. 2014 ).

These practices followed by the service providers, or the vendors could jeopardize the trust consumers have towards them. More research in this area would provide insights into the best social media marketing practices that are considered credible. Future research must focus on guiding marketers on ethical and credible practices in social media marketing and managing online reviews.

5.5 Research issue 5: presence of fake online reviews

Unlike incentivized reviews, deceptive opinion spams are written to sound real and to deceive the review readers (Ott, Cardie and Hancock 2013 ; Hernández Fusilier et al. 2015 ). Spammers use extreme language when it comes to praising or criticizing (Gao et al. 2021 ). These spammers are active on several social media and review platforms. As technology is continuously evolving deceptive opinion spam has found a way through the use of artificial intelligence. The social media platforms like Twitter and Facebook have experienced the rise of bot or automated accounts. This trend is even entering into online review systems and is a threat to the online review system Tousignant ( 2017 ). A study conducted by Yao et al. ( 2017 ) argues that the reviews generated by bots are not only undetectable but also scored as useful reviews. This is a serious issue as the whole purpose of online review platforms is to provide information that would lead an individual to make an informed decision, but these fake reviews severely damage the credibility of review site (Munzel 2016 ). In recent years, researchers started contributing to this area and have proposed models to detect fake reviews in different platforms such as app stores (Martens and Maalej 2019 ), online review platforms (Singh and Kumar 2017 ), and filtering fake reviews on TripAdvisor (Cardoso et al. 2018 ). However, presence of fake reviews can make the review users skeptical towards using the reviews. Future research must focus on the role of artificial intelligence in online review systems and its impact on consumers’ assessment of online review credibility. Research into tools to detect and curb the spread of fake reviews is needed to improve credibility of reviews.

5.6 Research issue 6: new forms of online reviews

Rapid technological developments have resulted in new digital formats of online reviews such as video and images. Past experimental design studies have primarily used stimuli in the form of textual reviews. As consumers use more and more multimedia data and engage in platforms such as Youtube.com or Instagram.com, research is required to examine the online review credibility and practices using new forms of reviews.

6 Theoretical contribution and managerial implications and conclusions

This paper makes three important theoretical contributions. First, it provides a consolidated account of antecedents, mediators and moderators of the construct online review credibility identifies five broad groups of antecedents. Second, this paper also makes a maiden attempt to map the antecedent themes to the theoretical frameworks in the literature. This mapping provides a holistic understanding of theories that examine various facets of online review credibility. In the process, we also identify theoretical lenses that are less investigated. Third we identify research gaps and issues that needs further investigation in the area of online review credibility. Some of the areas of future research include mitigation strategies for negative reviews and credibility of reviews in purchase of high-involvement product or service. Emergence of new forms of multimedia reviews, fake reviews and sponsored reviews have also triggered the need to push research beyond simple text reviews. Future research could use theoretical lens that have been less explored to investigate research issues in review credibility. There is a need to advance online review credibility research beyond the popular theoretical frameworks such as elaboration likelihood model, social influence theory, accessibility- diagnosticity theory, attribution theory, and theory of reasoned action.

The paper has several managerial implications. The lower credibility of reviews poses threat to its relevance in digital marketing and electronic commerce. Therefore, managers of electronic commerce must strive to adopt practices to preserve the trust and integrity of online reviews. Our review indicated five groups of antecedents of online review credibility: source characteristics, review characteristics, consumer characteristics, interpersonal characteristics in social media, and product type. Managers cannot control completely all the factors on the social media. However, by appropriately designing the e-commerce platform with the elements that influence credibility, managers will be able to improve their marketing communications. Awareness of review characteristics that impact review credibility would help managers to choose more appropriate measures to deal with negative and positive reviews. Managers must adopt a social media marketing strategy that is suitable to the context of the review and type of product.

Data availability

The dataset was generated by two licensed databases and thus cannot be made accessible.

Abedin E, Mendoza A, Karunasekera S (2021) Exploring the moderating role of readers’ perspective in evaluations of online consumer reviews. J Theor Appl Electron Commer Res 16:3406–3424. https://doi.org/10.3390/jtaer16070184

Article   Google Scholar  

Abu-El-Rub N, Minnich A, Mueen A (2017) Anomalous reviews owing to referral incentive. Proc 2017 IEEE/ACM int conf adv soc networks anal mining. ASONAM 2017:313–316. https://doi.org/10.1145/3110025.3110100

Anastasiei B, Dospinescu N, Dospinescu O (2021) Understanding the adoption of incentivized word-of-mouth in the online environment. J Theor Appl Electron Commer Res 16:992–1007. https://doi.org/10.3390/jtaer16040056

Bae S, Lee T (2011) Product type and consumers’ perception of online consumer reviews. Electron Mark 21:255–266. https://doi.org/10.1007/s12525-011-0072-0

Baharuddin NA, Yaacob M (2020) Dimensions of EWOM credibility on the online purchasing activities among consumers through social media. J Komun Malaysian J Commun 36:335–352

Bambauer-Sachse S, Mangold S (2010) The role of perceived review credibility in the context of brand equity dilution through negative product reviews on the internet. Adv Consum Res 18:38–45. https://doi.org/10.1016/j.jretconser.2010.09.003

Banerjee S, Chua AYK (2019) Trust in online hotel reviews across review polarity and hotel category. Comput Human Behav 90:265–275. https://doi.org/10.1016/j.chb.2018.09.010

Bansal HS, Voyer PA (2000) Word-of-mouth processes within a services purchase decision context. J Serv Res 3:166–177. https://doi.org/10.1177/109467050032005

Brand BM, Reith R (2022) Cultural differences in the perception of credible online reviews – the influence of presentation format. Decis Support Syst 154:113710. https://doi.org/10.1016/j.dss.2021.113710

Brand BM, Kopplin CS, Rausch TM (2022) Cultural differences in processing online customer reviews: holistic versus analytic thinkers. Electron Mark. https://doi.org/10.1007/s12525-022-00543-1

Brown JJ, Reingen PH (1987) Referral ties beav and word-of or *. J Consum Res 14:350–362. https://doi.org/10.1086/209118

Bughin J, Doogan J, Vetvik OJ (2010) A new way to measure word-of-mouth marketing. McKinsey Quarterly 2:113–6. https://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/a-new-way-to-measure-word-of-mouth-marketing . Accessed 13 March 2022

Camilleri AR (2017) The presentation format of review score information influences consumer preferences through the attribution of outlier reviews. J Interact Mark 39:1–14. https://doi.org/10.1016/j.intmar.2017.02.002

Cardoso EF, Silva RM, Almeida TA (2018) Towards automatic filtering of fake reviews. Neurocomputing 309:106–116. https://doi.org/10.1016/j.neucom.2018.04.074

Chakraborty U (2019) Perceived credibility of online hotel reviews and its impact on hotel booking intentions. Int J Contemp Hosp Manag 31:3465–3483. https://doi.org/10.1108/IJCHM-11-2018-0928

Chakraborty U, Bhat S (2017) The effects of credible online reviews on brand equity dimensions and its consequence on consumer behavior. J Promot Manag 24:57–82. https://doi.org/10.1080/10496491.2017.1346541

Chakraborty U, Bhat S (2018a) Credibility of online reviews and its impact on brand image. Manag Res Rev 41:148–164. https://doi.org/10.1108/MRR-06-2017-0173

Chakraborty U, Bhat S (2018b) Online reviews and its impact on brand equity. Int J Internet Mark Advert 12:159. https://doi.org/10.1504/ijima.2018.10011683

Chang HH, Wu LH (2014) An examination of negative e-WOM adoption: brand commitment as a moderator. Decis Support Syst 59:206–218. https://doi.org/10.1016/j.dss.2013.11.008

Chen Y, Xie J (2008) Online consumer review: word-of-mouth as a new element of marketing communication mix. Manage Sci 54:477–491. https://doi.org/10.1287/mnsc.1070.0810

Cheung CMK, Thadani DR (2012) The impact of electronic word-of-mouth communication: a literature analysis and integrative model. Decis Support Syst 54:461–470. https://doi.org/10.1016/j.dss.2012.06.008

Cheung M, Luo C, Sia C, Chen H (2009) Credibility of electronic word-of-mouth: informational and normative determinants of on-line consumer recommendations. Int J Electron Commer 13:9–38. https://doi.org/10.2753/JEC1086-4415130402

Cheung CMY, Sia CL, Kuan KKY (2012) Is this review believable? A study of factors affecting the credibility of online consumer reviews from an ELM perspective. J Assoc Inf Syst 13:618–635. https://doi.org/10.17705/1jais.00305

Chih WH, Wang KY, Hsu LC, Huang SC (2013) Investigating electronic word-of-mouth effects on online discussion forums: The role of perceived positive electronic word-of-mouth review credibility. Cyberpsychology, Behav Soc Netw 16:658–668. https://doi.org/10.1089/cyber.2012.0364

Chih WH, Hsu LC, Ortiz J (2020) The antecedents and consequences of the perceived positive eWOM review credibility. Ind Manag Data Syst 120:1217–1243. https://doi.org/10.1108/IMDS-10-2019-0573

Chiou JS, Hsiao CC, Su FY (2014) Whose online reviews have the most influences on consumers in cultural offerings? Professional vs consumer commentators. Internet Res 24:353–368. https://doi.org/10.1108/IntR-03-2013-0046

Chiou JS, Hsiao CC, Chiu TY (2018) The credibility and attribution of online reviews: differences between high and low product knowledge consumers. Online Inf Rev 42:630–646. https://doi.org/10.1108/OIR-06-2017-0197

Chu SC, Kim Y (2011) Determinants of consumer engagement in electronic word-of-mouth (eWOM) in social networking sites. Int J Advert 30:47–75. https://doi.org/10.2501/IJA-30-1-047-075

Collinger T, Malthouse E, Maslowska E, et al (2017) How online reviews influence sales. Evidence of the power of online reviews to shape customer behavior. In: Spiegel Res. Cent. https://spiegel.medill.northwestern.edu/how-online-reviews-influence-sales/%0A . Accessed 2 Nov 2021

Daowd A, Hasan R, Eldabi T et al (2020) Factors affecting eWOM credibility, information adoption and purchase intention on generation Y: a case from Thailand. J Enterp Inf Manag 34:838–859. https://doi.org/10.1108/JEIM-04-2019-0118

Doh SJ, Hwang JS (2009) How consumers evaluate eWOM (electronic word-of-mouth) messages. Cyberpsychol Behav 12:193–197. https://doi.org/10.1089/cpb.2008.0109

Dong Z (2015) How to persuade adolescents to use nutrition labels: effects of health consciousness, argument quality, and source credibility. Asian J Commun 25:84–101. https://doi.org/10.1080/01292986.2014.989241

Erkan I, Evans C (2016) The influence of eWOM in social media on consumers’ purchase intentions: an extended approach to information adoption. Comput Human Behav 61:47–55. https://doi.org/10.1016/j.chb.2016.03.003

Eslami SP, Ghasemaghaei M, Hassanein K (2018) Which online reviews do consumers find most helpful? A multi-method investigation. Decis Support Syst 113:32–42. https://doi.org/10.1016/j.dss.2018.06.012

Fang Y (2014) Beyond the credibility of electronic word of mouth : exploring eWOM adoption on social networking sites from affective and curiosity perspectives. Int J Electron Comm 18(3):67–102

Fang Y, Li C (2016) Electronic word-of-mouth on social networking sites : cue validity and cue utilization perspectives. Human Syst Manage 35:35–50. https://doi.org/10.3233/HSM-150853

Fisch C, Block J (2018) Six tips for your (systematic) literature review in business and management research. Manag Rev Q 68:103–106. https://doi.org/10.1007/s11301-018-0142-x

Flanagin AJ, Metzger MJ (2007) The role of site features, user attributes, and information verification behaviors on the perceived credibility of web-based information. New Media Soc 9:319–342. https://doi.org/10.1177/1461444807075015

Flanagin AJ, Metzger MJ (2013) Trusting expert versus user-generated ratings online: the role of information volume, valence, and consumer characteristics. Comput Human Behav 29:1626–1634. https://doi.org/10.1016/j.chb.2013.02.001

Floh A, Koller M, Zauner A (2013) Taking a deeper look at online reviews: the asymmetric effect of valence intensity on shopping behavior. J Mark Manag 29:646–670. https://doi.org/10.1080/0267257X.2013.776620

Fogg BJ, Marshall J, Laraki O, et al (2001) What makes web sites credible? A report on a large quantitative study. In proceedings of Conference on Human Factors in Computing Systems. Accessed 10 Jan 2021

Gao B, Hu N, Bose I (2017) Follow the herd or be myself? An analysis of consistency in behavior of reviewers and helpfulness of their reviews. Decis Support Syst 95:1–11. https://doi.org/10.1016/j.dss.2016.11.005

Gao Y, Gong M, Xie Y, Qin AK (2021) An attention-based unsupervised adversarial model for movie review spam detection. IEEE Trans Multimed 23:784–796. https://doi.org/10.1109/TMM.2020.2990085

Gilly MC, Graham JL, Wolfinbarger MF, Yale LJ (1998) A dyadic study of interpersonal information search. J Acad Mark Sci 26:83–100. https://doi.org/10.1177/0092070398262001

Gutt D, Neumann J, Zimmermann S et al (2019) Design of review systems – a strategic instrument to shape online reviewing behavior and economic outcomes. J Strateg Inf Syst 28:104–117

Guzzo T, Ferri F, Grifoni P (2022) What factors make online travel reviews credible? The consumers’ credibility perception-CONCEPT model. Societies. https://doi.org/10.3390/soc12020050

Gvili Y, Levy S (2016) Antecedents of attitudes toward eWOM communication: differences across channels. Internet Res 26:1030–1051. https://doi.org/10.1108/IntR-08-2014-0201

Ha EY, Lee H (2018) Projecting service quality: the effects of social media reviews on service perception. Int J Hosp Manag 69:132–141. https://doi.org/10.1016/j.ijhm.2017.09.006

Hernández Fusilier D, Montes-y-Gómez M, Rosso P, Guzmán Cabrera R (2015) Detecting positive and negative deceptive opinions using PU-learning. Inf Process Manag 51:433–443. https://doi.org/10.1016/j.ipm.2014.11.001

Herrero Á, Martín HS (2015) How online search behavior is influenced by user-generated content on review websites and hotel interactive websites. 27:1573–1597 https://doi.org/10.1108/IJCHM-05-2014-0255 .

Hong S, Pittman M (2020) eWOM anatomy of online product reviews : interaction effects of review number, valence, and star ratings on perceived credibility. Int J Advert. https://doi.org/10.1080/02650487.2019.1703386

Hsu LC, Chih WH, Liou DK (2016) Investigating community members’ eWOM effects in Facebook fan page. Ind Manag Data Syst 116:978–1004. https://doi.org/10.1108/IMDS-07-2015-0313

Hussain S, Guangju W, Jafar RMS et al (2018) Consumers’ online information adoption behavior: motives and antecedents of electronic word of mouth communications. Comput Human Behav 80:22–32. https://doi.org/10.1016/j.chb.2017.09.019

Işçi Ü, Kitapçi H (2020) Responses of Turkish consumers to product risk information in the context of negative EWOM. J Bus Econ Manag 21:1593–1609. https://doi.org/10.3846/jbem.2020.13383

Ismagilova E, Rana NP, Slade EL, Dwivedi YK (2020a) A meta-analysis of the factors affecting eWOM providing behaviour. Eur J Mark 55:1067–1102. https://doi.org/10.1108/EJM-07-2018-0472

Ismagilova E, Slade EL, Rana NP, Dwivedi YK (2020b) The effect of electronic word of mouth communications on intention to buy: a meta-analysis. Inf Syst Front 22:1203–1226. https://doi.org/10.1007/s10796-019-09924-y

Ismagilova E, Slade E, Rana NP, Dwivedi YK (2020c) The effect of characteristics of source credibility on consumer behaviour: a meta-analysis. J Retail Consum Serv 53:1–9. https://doi.org/10.1016/j.jretconser.2019.01.005

Izogo EE, Jayawardhena C, Karjaluoto H (2022) Negative eWOM and perceived credibility: a potent mix in consumer relationships. Int J Retail Distrib Manag. https://doi.org/10.1108/IJRDM-01-2022-0039

Jha AK, Shah S (2021) Disconfirmation effect on online review credibility: an experimental analysis. Decis Support Syst 145:113519. https://doi.org/10.1016/j.dss.2021.113519

Jiménez FR, Mendoza NA (2013) Too popular to ignore : the influence of online reviews on purchase intentions of search and experience products. J Interact Mark 27:226–235. https://doi.org/10.1016/j.intmar.2013.04.004

Johnson TJ, Kaye BK (2016) Some like it lots: the influence of interactivity and reliance on credibility. Comput Human Behav 61:136–145. https://doi.org/10.1016/j.chb.2016.03.012

Klaus T, Changchit C (2019) Toward an understanding of consumer attitudes on online review usage. J Comput Inf Syst 59:277–286. https://doi.org/10.1080/08874417.2017.1348916

Kozinets RV (2016) Amazonian forests and trees: Multiplicity and objectivity in studies of online consumer-generated ratings and reviews, a commentary on de Langhe, Fernbach, and Lichtenstein. J Consum Res 42:834–839. https://doi.org/10.1093/jcr/ucv090

Kuckertz A, Brändle L (2022) Creative reconstruction: a structured literature review of the early empirical research on the COVID-19 crisis and entrepreneurship. Manag Rev Q 72:281–307. https://doi.org/10.1007/s11301-021-00221-0

Kusumasondjaja S, Shanka T, Marchegiani C (2012) Credibility of online reviews and initial trust: the roles of reviewer’s identity and review valence. J Vacat Mark 18:185–195. https://doi.org/10.1177/1356766712449365

Liberati A, Altman DG, Tetzlaff J, et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 62:e1–e34. https://doi.org/10.1016/j.jclinepi.2009.06.006

Lee KT, Koo DM (2012) Effects of attribute and valence of e-WOM on message adoption: moderating roles of subjective knowledge and regulatory focus. Comput Human Behav 28:1974–1984. https://doi.org/10.1016/j.chb.2012.05.018

Lee J, Park DH, Han I (2011) The different effects of online consumer reviews on consumers’ purchase intentions depending on trust in online shopping malls: an advertising perspective. Internet Res 21:187–206. https://doi.org/10.1108/10662241111123766

Lim Y, Van Der Heide B (2015) Evaluating the wisdom of strangers: the perceived credibility of online consumer reviews on yelp. J Comput Commun 20:67–82. https://doi.org/10.1111/jcc4.12093

Lin C, Wu Y-S, Chen J-CV (2013) Electronic word-of-mouth: the moderating roles of product involvement and brand image. In proceedings of 2013 international conference on technology innovation and industrial management. Accessed 14 Jan 2021

Linnenluecke MK, Marrone M, Singh AK (2020) Conducting systematic literature reviews and bibliometric analyses. Aust J Manag 45:175–194. https://doi.org/10.1177/0312896219877678

Liu W, Ji R (2018) Examining the role of online reviews in Chinese online group buying context: the moderating effect of promotional marketing. Soc Sci 7:141–157. https://doi.org/10.3390/socsci7080141

Lo AS, Yao SS (2018) What makes hotel online reviews credible?: An investigation of the roles of reviewer expertise, review rating consistency and review valence. Int J Contemp Hosp Manag 31:41–60. https://doi.org/10.1108/IJCHM-10-2017-0671

Luo C, Wu J, Shi Y, Xu Y (2014) The effects of individualism-collectivism cultural orientation on eWOM information. Int J Inf Manage 34:446–456. https://doi.org/10.1016/j.ijinfomgt.2014.04.001

Luo C, Luo X, Xu Y et al (2015) Examining the moderating role of sense of membership in online review evaluations. Inf Manag 52:305–316. https://doi.org/10.1016/j.im.2014.12.008

Majali T, Alsoud M, Yaseen H et al (2022) The effect of digital review credibility on Jordanian online purchase intention. Int J Data Netw Sci 6:973–982. https://doi.org/10.5267/j.ijdns.2022.1.014

Manganari EE, Dimara E (2017) Enhancing the impact of online hotel reviews through the use of emoticons. Behav Inf Technol 36:674–686. https://doi.org/10.1080/0144929X.2016.1275807

Mariani MM, Borghi M, Kazakov S (2019) The role of language in the online evaluation of hospitality service encounters: an empirical study. Int J Hosp Manag 78:50–58. https://doi.org/10.1016/j.ijhm.2018.11.012

Martens D, Maalej W (2019) Towards understanding and detecting fake reviews in app stores. Empir Softw Eng 24:3316–3355. https://doi.org/10.1007/s10664-019-09706-9

Mayzlin D, Dover Y, Chevalier J (2014) Promotional reviews: an empirical investigation of online review manipulation. Am Econom Rev 104(8):2421–2455

Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:332–336. https://doi.org/10.1136/bmj.b2535

Moran G, Muzellec L (2017) eWOM credibility on social networking sites: a framework. J Mark Commun 23:149–161. https://doi.org/10.1080/13527266.2014.969756

Mudambi SM, Schuff D, Schuff D (2010) What makes a helpful online review? A study of customer reviews on amazon. com. MIS Q 34:185–200

Mumuni AG, O’Reilly K, MacMillan A et al (2019) Online product review impact: the relative effects of review credibility and review relevance. J Int Commer 19:153–191. https://doi.org/10.1080/15332861.2019.1700740

Munzel A (2016) Journal of retailing and consumer services assisting consumers in detecting fake reviews : the role of identity information disclosure and consensus. J Retail Consum Serv 32:96–108. https://doi.org/10.1016/j.jretconser.2016.06.002

Nadkarni S, Prügl R (2021) Digital transformation: a review, synthesis and opportunities for future research. Manag Rev Q 71:233–341. https://doi.org/10.1007/s11301-020-00185-7

Neumann T (2021) The impact of entrepreneurship on economic, social and environmental welfare and its determinants: a systematic review. Manag Rev Q 71:553–584. https://doi.org/10.1007/s11301-020-00193-7

Newell SJ, Goldsmith RE (2001) The development of a scale to measure perceived corporate credibility. J Bus Res 52:235–247. https://doi.org/10.1016/S0148-2963(99)00104-6

Niu W, Huang L, Li X et al (2022) Beyond the review information: an investigation of individual- and group-based presentation forms of review information. Inf Technol Manag. https://doi.org/10.1007/s10799-022-00361-z

Okoli C (2015) A guide to conducting a standalone systematic literature review. Commun Assoc Inf Syst 37:879–910. https://doi.org/10.17705/1cais.03743

Ott M, Cardie C, Hancock JT (2013) Negative deceptive opinion spam. NAACL HLT 2013.In: proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies 497–501

Park DH, Lee J (2008) eWOM overload and its effect on consumer behavioral intention depending on consumer involvement. Electron Commer Res Appl 7:386–398. https://doi.org/10.1016/j.elerap.2007.11.004

Pee LG (2016) Negative online consumer reviews: can the impact be mitigated? Int J Mark Res 58:545–568. https://doi.org/10.2501/IJMR-2016-035

Pentina I, Basmanova O, Sun Q (2017) Message and source characteristics as drivers of mobile digital review persuasiveness: does cultural context play a role? Int J Internet Mark Advert 11:1–21

Google Scholar  

Pentina I, Bailey AA, Zhang L (2018) Exploring effects of source similarity, message valence, and receiver regulatory focus on yelp review persuasiveness and purchase intentions. J Mark Commun 24:125–145. https://doi.org/10.1080/13527266.2015.1005115

Petty RE, Cacioppo JT, Schumann D (1983) Central and peripheral routes to advertising effectiveness: the moderating role of involvement. J Consum Res 10:135. https://doi.org/10.1086/208954

Qahri-Saremi H, Montazemi AR (2019) Factors affecting the adoption of an electronic word of mouth message: a meta-analysis. J Manag Inf Syst 36:969–1001. https://doi.org/10.1080/07421222.2019.1628936

Qiu L, Pang J, Lim KH (2012) Effects of conflicting aggregated rating on eWOM review credibility and diagnosticity: the moderating role of review valence. Decis Support Syst 54:631–643. https://doi.org/10.1016/j.dss.2012.08.020

Ran L, Zhenpeng L, Bilgihan A, Okumus F (2021) Marketing China to U.S travelers through electronic word-of-mouth and destination image: taking Beijing as an example. J Vacat Mark. https://doi.org/10.1177/1356766720987869

Reyes-Menendez A, Saura JR, Martinez-Navalon JG (2019) The Impact of e-WOM on hotels management reputation: exploring tripadvisor review credibility with the ELM model. IEEE Access 7:68868–68877. https://doi.org/10.1109/ACCESS.2019.2919030

Schiffman LG, Kanuk LL (2000) Consumer behavior, 7th edn. Prentice-Hall, Upper Saddle River, NJ

Seghers M, De Clerck B, Lybaert C (2021) When form deviates from the norm: attitudes towards old and new vernacular features and their impact on the perceived credibility and usefulness of Facebook consumer reviews. Lang Sci 87:101413. https://doi.org/10.1016/j.langsci.2021.101413

Senecal S, Nantel J (2004) The influence of online product recommendations on consumers’ online choices. J Retail 80:159–169. https://doi.org/10.1016/j.jretai.2004.04.001

Shamhuyenhanzva RM, van Tonder E, Roberts-Lombard M, Hemsworth D (2016) Factors influencing generation Y consumers’ perceptions of eWOM credibility: a study of the fast-food industry. Int Rev Retail Distrib Consum Res 26:435–455. https://doi.org/10.1080/09593969.2016.1170065

Shin E, Chung T, Damhorst ML (2020) Are negative and positive reviews regarding apparel fit influential? J Fash Mark Manag 25:63–79. https://doi.org/10.1108/JFMM-02-2020-0027

Shukla A, Mishra A (2021) Effects of visual information and argument concreteness on purchase intention of consumers towards online hotel booking. Vision. https://doi.org/10.1177/09722629211038069

Shukla A, Mishra A (2022) Role of review length, review valence and review credibility on consumer’s online hotel booking intention. FIIB Bus Rev. https://doi.org/10.1177/23197145221099683

Siddiqui MS, Siddiqui UA, Khan MA et al (2021) Creating electronic word of mouth credibility through social networking sites and determining its impact on brand image and online purchase intentions in India. J Theor Appl Electron Commer Res 16:1008–1024. https://doi.org/10.3390/jtaer16040057

Singh M, Kumar L S (2017) Model for detecting fake or spam reviews. In: advances in intelligent systems and computing pp 213–217. Accessed 05 Jan 2022

Smith KT (2011) Digital marketing strategies that Millennials find appealing, motivating, or just annoying. J Strateg Mark 19:489–499. https://doi.org/10.1080/0965254X.2011.581383

Snyder H (2019) Literature review as a research methodology: an overview and guidelines. J Bus Res 104:333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

Thomas MJ, Wirtz BW, Weyerer JC (2019) Determinants of online review credibility and its impact on consumers’ purchase intention. J Electron Commer Res 20:1–20

Tien DH, Amaya Rivas AA, Liao YK (2018) Examining the influence of customer-to-customer electronic word-of-mouth on purchase intention in social networking sites. Asia Pacific Manag Rev 24:238–249. https://doi.org/10.1016/j.apmrv.2018.06.003

Tousignant L (2017) Robots learned how to write fake Yelp reviews like a human. New York Post. https://nypost.com/2017/08/31/robots-learned-how-to-write-fake-yelp-reviews-like-a-human/ . Accessed 10 Jan 2021

Tran VD, Can TK (2020) Factors affecting the credibility of online reviews on tiki: An assessment study in vietnam. Int J Data Netw Sci 4:115–126. https://doi.org/10.5267/j.ijdns.2020.2.005

Tsao WC, Hsieh MT (2015) eWOM persuasiveness: do eWOM platforms and product type matter? Electron Commer Res 15:509–541. https://doi.org/10.1007/s10660-015-9198-z

Tseng S, Fogg BJ (1999) Credibility and computing technology. Commun ACM 42:39

Tversky A, Kahneman D (1991) Loss Aversion in Riskless Choice. Q J Econ 106:1039–1061

v. Wangenheim F, Bayón T, (2004) The effect of word of mouth on services switching. Eur J Mark 38:1173–1185. https://doi.org/10.1108/03090560410548924

Van Lohuizen AW, Trujillo-Barrera A (2019) The influence of online reviews on restaurants: the roles of review valence, platform, and credibility. J Agric Food Ind Organ. https://doi.org/10.1515/jafio-2018-0020

Vendemia MA (2017) (Re)viewing reviews: effects of emotionality and valence on credibility perceptions in online consumer reviews. Commun Res Reports 34:230–238. https://doi.org/10.1080/08824096.2017.1286470

Verma D, Dewani PP (2020) eWOM credibility: a comprehensive framework and literature review. Online Inf Rev 45:481–500. https://doi.org/10.1108/OIR-06-2020-0263

Walter AT (2021) Organizational agility: ill-defined and somewhat confusing? A systematic literature review and conceptualization. Manag Rev Q 71:343–391. https://doi.org/10.1007/s11301-020-00186-6

Wang Y, Chan SCF, Ngai G, Leong HV (2013) Quantifying reviewer credibility in online tourism. Lect Notes Comput Sci 8055:381–395. https://doi.org/10.1007/978-3-642-40285-2_33

Wang X, Xu F, Luo X, (Robert), Peng L, (2022) Effect of sponsorship disclosure on online consumer responses to positive reviews: the moderating role of emotional intensity and tie strength. Decis Support Syst 156:113741. https://doi.org/10.1016/j.dss.2022.113741

Weisfeld-Spolter S, Sussan F, Gould S (2014) An integrative approach to eWOM and marketing communications. Corp Commun 19:260–274. https://doi.org/10.1108/CCIJ-03-2013-0015

Wu K, Noorian Z, Vassileva J, Adaji I (2015) How buyers perceive the credibility of advisors in online marketplace: review balance, review count and misattribution. J Trust Manag. https://doi.org/10.1186/s40493-015-0013-5

Xia M, Huang Y, Duan W, Whinston AB (2009) Ballot box communication in online communities. Commun ACM. https://doi.org/10.1145/1562164.1562199

Xiang Z, Du Q, Ma Y, Fan W (2017) A comparative analysis of major online review platforms: implications for social media analytics in hospitality and tourism. Tour Manag 58:51–65. https://doi.org/10.1016/j.tourman.2016.10.001

Xu Q (2014) Should i trust him? the effects of reviewer profile characteristics on eWOM credibility. Comput Human Behav 33:136–144. https://doi.org/10.1016/j.chb.2014.01.027

Xue F, Zhou P (2010) The effects of product involvement and prior experience on chinese consumers’ responses to online word of mouth. J Int Consum Mark 23:45–58. https://doi.org/10.1080/08961530.2011.524576

Yan L, Hua C (2021) Which reviewers are honest and caring? The effect of constructive and prosocial information on the perceived credibility of online reviews. Int J Hosp Manag 99:102990. https://doi.org/10.1016/j.ijhm.2021.102990

Yan Q, Wu S, Wang L et al (2016) E-WOM from e-commerce websites and social media: which will consumers adopt? Electron Commer Res Appl 17:62–73. https://doi.org/10.1016/j.elerap.2016.03.004

Yang J, Mai ES (2010) Experiential goods with network externalities effects: an empirical study of online rating system. J Bus Res 63:1050–1057. https://doi.org/10.1016/j.jbusres.2009.04.029

Yao Y, Viswanath B, Cryan J et al (2017) Automated crowdturfing attacks and defenses in online review systems. Proc ACM Conf Comput Commun Secur. https://doi.org/10.1145/3133956.3133990

You Y, Vadakkepatt GG, Joshi AM (2015) A meta-analysis of electronic word-of-mouth elasticity. J Mark 79:19–39. https://doi.org/10.1509/jm.14.0169

Zha X, Li J, Yan Y (2015) Advertising value and credibility transfer: attitude towards web advertising and online information acquisition. Behav Inf Technol 34:520–532. https://doi.org/10.1080/0144929X.2014.978380

Zhang X, Wu Y, Wang W (2020) eWOM, what are we suspecting? Motivation, truthfulness or identity. J Inform, Commun Ethics Soc. https://doi.org/10.1108/JICES-12-2019-0135

Zheng L (2021) The classification of online consumer reviews: a systematic literature review and integrative framework. J Bus Res 135:226–251. https://doi.org/10.1016/j.jbusres.2021.06.038

Download references

Open access funding provided by Manipal Academy of Higher Education, Manipal. The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and affiliations.

Manipal Institute of Management, Manipal Academy of Higher Education, Manipal, Karnataka, India

T A Pai Management Institute, Manipal Academy of Higher Education, Manipal, Karnataka, India

Pallavi Upadhyaya

You can also search for this author in PubMed   Google Scholar

Contributions

PK: Writing—Original Draft, Methodology, Formal analysis, Investigation. PU: Conceptualization; Validation, Data Curation, Writing—Review & Editing, Visualization, Supervision.

Corresponding author

Correspondence to Pallavi Upadhyaya .

Ethics declarations

Conflict of interest.

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Pooja, K., Upadhyaya, P. What makes an online review credible? A systematic review of the literature and future research directions. Manag Rev Q (2022). https://doi.org/10.1007/s11301-022-00312-6

Download citation

Received : 01 March 2022

Accepted : 21 November 2022

Published : 05 December 2022

DOI : https://doi.org/10.1007/s11301-022-00312-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Review credibility
• Systematic literature review
• Online review
• eWOM credibility

JEL Classification

• Find a journal
• Publish with us
• Track your research

• Military & Veterans
• Transfer & Credits
• Education Partnerships
• COVID-19 Info
• 844-PURDUE-G
• Student Login
• Request Info
• Bachelor of Science
• Master of Science
• Associate of Applied Science
• Graduate Certificate
• Master of Business Administration
• ExcelTrack Master of Business Administration
• ExcelTrack Bachelor of Science
• Postbaccalaureate Certificate
• Certificate
• Associate of Applied Science (For Military Students)
• Programs and Courses
• Master of Public Administration
• Doctor of Education
• Postgraduate Certificate
• Bachelor of Science in Psychology
• Master of Health Care Administration
• Master of Health Informatics
• Doctor of Health Science
• Associate of Applied of Science (For Military Students)
• Associate of Science (For Military Students)
• Master of Public Health
• Executive Juris Doctor
• Juris Doctor
• Dual Master's Degrees
• ExcelTrack Master of Science
• Master of Science (DNP Path)
• Bachelor of Science (RN-to-BSN)
• ExcelTrack Bachelor of Science (RN-to-BSN)
• Associate of Science
• Doctor of Nursing Practice
• Master of Professional Studies

The average Purdue Global military student is awarded 54% of the credits needed for an associate's and 45% of the credits needed for a bachelor's.

• General Education Mobile (GEM) Program
• AAS in Health Science
• AS in Health Science
• BS in Organizational Management
• BS in Professional Studies
• AAS in Criminal Justice
• AAS in Small Group Management
• AAS Small Group Management
• Master's Degrees
• Bachelor's Degrees
• Associate's Degrees
• Certificate Programs
• Noncredit Courses
• Tuition and Financial Aid Overview
• Financial Aid Process
• Financial Aid Awards
• Financial Aid Resources
• Financial Aid Frequently Asked Questions
• Financial Aid Information Guide
• Tuition and Savings
• Aviation Degree Tuition and Fees
• Professional Studies Tuition and Fees
• Single Courses and Micro-Credentials
• Time and Tuition Calculator
• Net Price Calculator
• Military Benefits & Tuition Assistance
• Military Educational Resources
• Military Tuition Reductions
• Military Spouses
• Student Loans
• Student Grants
• Outside Scholarships
• Loan Management
• Financial Literacy Tools
• Academic Calendar
• Admissions Process
• General Requirements
• Technology Requirements
• Transfer Students
• Alternative Credit Center
• DREAMers Education Initiative
• Student Identity
• Student Experience
• Online Experience
• Student Life
• Alumni Engagement
• International Students
• Academic Support
• All Purdue Online Degrees
• Career Services
• COVID-19 FAQs
• Student Accessibility Services
• Student Resources
• Transcript Request
• About Purdue Global
• Accreditation
• Approach to Learning
• Career Opportunities
• Diversity Initiatives
• Purdue Global Commitment
• Cybersecurity Center
• Chancellor's Corner
• Purdue Global Moves
• Leadership and Board
• Facts and Statistics
• Researcher Request Intake Form

Most Commonly Searched:

• All Degree Programs
• Communication
• Criminal Justice
• Fire Science
• Health Sciences
• Human Services
• Information Technology
• Legal Studies
• Professional Studies
• Psychology and ABA
• Public Policy
• Military and Veterans
• Tuition & Financial Aid Overview
• Tuition and Fee Finder
• Financial Aid FAQs
• Military Benefits & Aid
• Admissions Overview
• Student Experience Overview
• Academic Support Overview
• Online Learning

Check Your Sources: A Checklist for Validating Academic Information

A virtual flood of information is available with just a few clicks, but it is important to remember that abundance does not mean quality. There are plenty of inaccurate articles and misinformation online, making it crucial to fully understand how to discern the credibility of sources. Although the ability to validate information is always important, it is especially vital for students as they pursue information for academic research and papers.

This article provides a comprehensive checklist that can help you weed out bad information and find reliable and accurate sources for your academic writing and research endeavors.

Why Credibility Matters in Academic Research

It is easy to understand why credibility matters; after all, it is the cornerstone of academic research. The implications of being credible, however, extend beyond grades and academia.

Reliable sources lend weight to arguments, ensuring they stand up to scrutiny. Conversely, unreliable sources can introduce errors into a field of study, leading to flawed conclusions. This type of situation can affect the integrity of the broader knowledge base and adversely affect the researcher's reputation.

A Checklist for Validating Academic Information

As information continues to proliferate, the ability to distinguish credible from questionable becomes increasingly important. This checklist offers a structured approach to ensure your research is grounded in authoritative and relevant sources, bolstering the integrity of your work.

1. Identify Who Provided the Information

The credibility of information often hinges on the expertise and reputation of its provider.

Author credentials: A source's reliability often heavily relies on the expertise of its author. When looking at sources, check the author’s academic background and look for additional publications credited to them.

Institutional affiliation: Reputable institutions typically adhere to rigorous publication standards. If a source comes from a recognized university or research body, it's likely undergone thorough review. This is not foolproof, but it serves as a green flag for the reliability of the source.

Peer review: In academia, peer review is the gold standard. It means other experts in the field have examined and approved the content. You can usually find this information in the editorial guidelines for the journal or website that published the content.

2. Acknowledge Any Potential Bias

Every piece of information carries a perspective, so it is crucial to discern its objectivity before using it as a source.

Objective vs. subjective: While no source is entirely free from bias, it is vital to distinguish between objective research and opinion pieces. The former is based on empirical evidence, while the latter reflects personal viewpoints.

Funding sources: Research funded by organizations with vested interests might be skewed. Always check the acknowledgments or disclosure section.

Affiliations: Authors affiliated with certain groups might have inherent biases. It does not invalidate their work, but you should be aware of it so you can determine if the information is credible or overly biased.

3. Identify Claims Made Without Proper Data

Valid academic claims are rooted in evidence, making it essential to scrutinize the data backing them.

Evidence-based claims: In academic research, claims should be backed by data. If a source makes broad assertions without evidence, approach it with caution.

Transparent methodology: A credible source will detail its methodology, allowing others to replicate the study or understand its basis.

Unsupported statements: Be wary of sweeping claims that do not reference other studies or data. This is a red flag that indicates the information may not be credible.

4. Check the Purpose of the Information

Understanding the intent behind a source helps in assessing its relevance and potential bias.

Informative vs. persuasive: Is the source aiming to inform based on evidence, or is it trying to persuade? Both can be valid, but it is essential to know the difference and decide if the information is usable on a case-by-case basis.

Primary vs. secondary sources: Primary sources offer direct evidence or firsthand testimony. Secondary sources analyze or interpret primary sources. While both types of sources can be credible, you should still understand and distinguish between them.

Audience and conflicts: Consider who the intended audience is because this can shape the type of information being shared. A paper written for industry professionals might have a different tone and depth than one written for general readers.

5. Check Publication Dates

The age of a source can influence its relevance and applicability to current research in several key ways.

Relevance and recency: In quickly evolving fields, recent publications are crucial, as they reflect the latest findings and consensus. However, this does not mean older sources are obsolete. They can offer foundational knowledge or a historical perspective. It is just important to be aware of the dates associated with all information you plan on using.

Historical context: When citing older sources, it is essential to understand their context. How has the field evolved since then? Are the findings still relevant and accurate, or has newer research superseded them?

Topic evolution: Using older sources can provide unique insight. Tracking the progression of thought on a subject can provide depth to your research, showing how current perspectives were shaped.

6. Assess the Source's Reputation

A source's standing in the academic community can be a strong indicator of its reliability.

Citations: If a source is frequently cited in other works, it is a positive indication — but not a fool-proof test. The reputation and authority of where the citation occurs can also reflect on its credibility.

Retractions/corrections: Check if the source has any associated retractions or corrections. This might indicate issues with the content but may also indicate dedication to sharing accurate information.

7. Verify Citations and References

Reliable academic work builds upon previous research, making citations a key component of credibility.

Backed claims: Ensure that the source's claims are supported by credible references. These should be easy to find, easy to access, and not outdated.

Authenticity of citations: Check the original studies or data cited to ensure they have been represented accurately. You should never rely on a source’s representation of facts but rather check them against the originating source.

Self-citation: While authors will sometimes cite their previous work, excessive self-citation can be a red flag.

Additional Tips on How to Know if a Source Is Credible

Consult experts: If you are unsure about a source, reach out to experts or professors in the field. Their experience can provide insights into the source's reliability.

Check for comprehensive coverage: Reliable sources often cover topics in depth, addressing multiple facets of an issue rather than presenting a one-sided view.

Examine the writing style: Credible sources typically maintain a professional tone, avoiding sensationalism or overly emotional language. Spelling and grammar errors are a red flag.

Look for transparency: Trustworthy sources are transparent about their research methods, data collection, and any potential conflicts of interest.

In academic writing, the strength of your work is deeply rooted in the credibility of your sources. By carefully evaluating your sources, you can ensure that you're presenting accurate information that stands up to scrutiny. This process starts with systematically validating the information you find for bias, outdated information, unsupported claims, and many other criteria. In the end, however, it is your discernment that keeps unscrupulous information from ending up in your research. 

Purdue Global Offers Robust Resources to Enhance Academic Research

At Purdue Global, students have access to academic tutoring and support centers to assist them in their efforts. If you are ready to kick-start your academic journey or have questions, reach out today for more information about the 175+ online degree and certificate programs from Purdue Global.

About the Author

Purdue Global

Earn a degree you're proud of and employers respect at Purdue Global, Purdue's online university for working adults. Accredited and online, Purdue Global gives you the flexibility and support you need to come back and move your career forward. Choose from 175+ programs, all backed by the power of Purdue.

• General Education
• Legal Studies & Public Policy

Your Path to Success Begins Here

Learn more about online programs at Purdue Global and download our program guide.

Connect with an Advisor to explore program requirements, curriculum, credit for prior learning process, and financial aid options.

Researching and Writing a Paper: Reliability of Sources

• Outline Note-Taking
• Summarizing
• Bibliography / Annotated Bibliography
• Thesis Sentences
• Ideas for Topics
• The Big List of Databases and Resource Sources
• Keywords and Controlled Vocabulary
• Full Text Advice
• Database Searching Videos!
• How to Read a Scholarly Article.
• Citation Styles
• Citation Videos!
• Citation Tips & Tricks
• Videos about Evaluating Sources!
• Unreliable Sources and 'Fake News'
• An Outline for Writing!
• Formatting your paper!

Techniques for Evaluating Resources

No matter how good the database you search in is, or how reliable an information website has been in the past, you need to evaluate the sources you want to use for credibility and bias before you use them*. (You may also want to spot fake news as you browse the Internet or other media - not all fake news is online).

This page discusses eight different tools for evaluating sources (there are so many different tools because evaluating the reliability or quality of information is an important topic, because there are many ways to look at the topic, and every librarian wants to help you succeed). Look through these approaches and use the approaches or combinations of approaches that work for you. The tools are:

5Ws (and an H)

A.S.P.E.C.T.

• Evaluating Research Articles
• Lateral Reading  ("what do other sources say?")

The CRAAP Test

We also have a variety of videos about evaluating sources available for your learning and entertainment: Click Here !

* Note: a biased source - and technically most sources are biased - can be a useful source as long as you understand what the bias or biases are. A source that is trying to be reliable will often identify some or all of its biases. (Every person has a limited perspective on the events they observe or participate in, and most of the time their perceptions are influenced by assumptions they may not be aware of. So, even when you have some really solid reasons to trust a source as 100% reliable and accurate, be alert for 'what is not mentioned' and for what biases there might be [this is particularly tricky when you share those biases, and why reviewing your research/paper/presentation with someone else is always a good idea]).

The 5Ws and an H.

Back to the top of the page

• Who are the authors or creators?
• What are their credentials? Can you find something out about them in another place?
• Who is the publisher or sponsor?
• Are they reputable?
• What is the publisher’s interest (if any) in this information?
• If it's from a website, does it have advertisements?
• Is this fact or opinion?
• Is it biased? Can you still use the information, even if you know there is bias?
• Is the the site trying to sell you something, convert you to something, or make you vote for someone?
• What kind of information is included in the resource?
• Is content of the resource primarily opinion? Is is balanced?
• Is it provided for a hobbiest, for entertainment, or for a serious audience?
• Does the creator provide references or sources for data or quotations?
• How recent is the information?
• Is it current enough for your topic?
• If the information is from a website, when was the site last updated?

Authority Information resources are a product of their creator's expertise and reliability, and are evaluated based on the information need and the context in which the information will be used. Authority is constructed: various communities often recognize different types of authority (knowledge, accuracy). Authority is contextual because you may need additional information to help determine the accuracy or comprehensiveness, and the sort of authority the source contains. (Writing a paper about ' urban myths ' requires different sorts of authority than writing a paper disproving an urban myth.)

Using this concept means you have to identify the different types of authority that might be relevant, and why the author considers themselves reliable, as well as why their community considers them reliable. An author can be a person, journalist, scholar, organization, website, etc. Author is different from authority, authority is the quality that gives an author trustworthiness....and not all authors have the same trustworthiness.

Evaluating research articles

Evaluating research articles: Evaluating evidence-based research articles in scholarly journals requires deep knowledge of the discipline, which you might not acquire until you are deeper into your education. These guiding questions can help you evaluate a research report, even if you are not an expert in the field. Questions include:

• Why was the study undertaken? The aim of the research may be intended to generate income, lobby for policy changes, evaluate the impact of a program, or create a new theory.  These variations in intent influence the research questions, the data collection, the analysis, and how the results are presented. To make best use of the findings for your purposes, you should keep the intent of the study in mind.
• Who conducted the study? It is important to look at who conducted the research, and if the organization or individual in question has the expertise required for conducting research on the topic. Looking to see if the organization is interested in a specific research outcome is also a good practice. The research should be clear about how the different stages of the study were conducted to guarantee its objectivity.
• Who funded the research? It is equally important to look at who sponsored or funded the study because this sometimes affects the objectivity or accuracy of the study. (If, for example, a soap-maker sponsors a study on the efficiency of different soaps, you should be critical of the results, particularly if their brand of soap is the best at cleaning.)
• How was the data collected? In the social sciences, structured interviews and self-completion questionnaires are perhaps the two most common ways of collecting quantitative data. How the people in the study were recruited is essential for determining how representative the results are. (There are two main types of samples, probability and non-probability samples. A probability sample is one in which every individual in the population has the same chance of being included in the study. It is also a prerequisite for being able to generalize the findings to the population. Pretend you survey first-year students by asking student clubs to share the survey on their social media. This non-probability snowball sample is more likely to reach students active in the clubs, therefore the results will not be representative of, or generalizable to, all students.)
• Is the sample size and response rate sufficient? The bigger the sample size the greater the chance that the results are accurate. After a sample size of around 1000, gains in accuracy become less significant. However, limited time and money often make such a large sample not practical. The similarity of the population also affects the desired sample size; a more diverse population requires a larger sample to sufficiently include the different parts of the population. The response rate is a complementary measure to the sample size, showing how many of the suitable individuals in the sample have provided a usable response. (In web surveys, response rates tend to be lower than in other types of surveys, and are therefore less accurate.)
• Does the research make use of secondary data? Data can be collected for the purposes of the study or existing data gathered for a different study can be used. If existing data sets collected for another study are used, reflecting on how usable that data is for the newer study is important.
• Does the research measure what it claims to measure? A commonly used word in statistics to describe the trustworthiness of research is ‘validity’. Validity refers to the extent to which an assumption or measurement is consistent with reality. Does it measure what it intends to measure? For example, a study investigates gender discrimination of faculty and looks at the number of cases of discrimination presented by female faculty. But, if the study does not look at the reason for these discrimination complaints (gender, ethnicity, religion, age, sexual orientation, etc.) it cannot be assumed that gender discrimination either increased or decreased.
• Can the findings be generalized to my situation? There is often a tendency to generalize research findings. Two key standards have to be met to do this. First, results apply only to the population of the study. Second, data must be collected via a probability sample, i.e. everyone eligible to be in the study has the same chance of being included in the study. Too often papers do not discuss many of the aspects of the data collection and analysis. Transparently and clearly describing how the research was conducted is essential for the reader to understand the trustworthiness of the research paper in their hands.

Lateral Reading

The Internet has democratized access to information, but the Internet has also been filled with a flood of misinformation, fake news, propaganda, and idiocy, presented as objective analysis. Since any single source is suspect, fact checkers read laterally.  They leave a site in its tab after a quick look around and open up new browser tabs in order to judge the credibility of the original site.

Lateral reading is the process of verifying what you are reading while you are reading it. It allows you to read deeply and broadly while gaining a fuller understanding of an issue or topic and determining whether, or how much, to trust the content as presented.

Vertical reading occurs when the reader simply reads the article or site without going further, assuming that if it ‘looks reliable’ it is reliable. The reader may use some superficial evaluation strategies to determine if the site is credible, such as reading the ‘about’ page, looking at its URL extension (.edu, .org, .com, .gov, etc.), or assessing its advertising. A good start, but there is much more to look at:

• Determine the author's reliability, intents, and biases, by searching for articles by other writers on the same topic (and also looking for other articles by that same author).
• Understand the perspective of the site's analyses. (What are they assuming, what do they want you to assume?)
• Determine whether the site has an editorial process or expert reputation supporting the reliability and accuracy of its content.

Use the strategies and ask the questions that professional fact-checkers use:

• Go beyond the "about" page of the site you are reading.
• Search for articles by other writers on the same topic.
• Search for articles about the site/publication you are reading (and/or articles about the authors featured on the site).

Ask the following:

• Who funds or sponsors the site where the original piece was published? (And who funds/sponsors the site you found the article at?) 
• What do other authoritative sources have to say about that site and that topic?
• When you do a search on the topic of the original piece, are the initial results from fact-checking organizations? (If so, what do they say?)
• Have questions been raised about other articles the author has written or that have appeared on that site?
• Does what you are finding elsewhere contradict the original piece? (If there are contradictions, what is the reliability of those contradicting sites?)

Are reliable news outlets reporting on (or perhaps more important, not reporting on) what you are reading? (Does why reliable news outlets are or are-not reporting on the topic increase or decrease the reliability of the site you are assessing?)

Sometimes the 'good answer' to the above questions is a 'yes', sometimes a 'no', and sometimes 'it's complicated'. Reliable and unreliable sources are everywhere in the information we have access to - some sources are rarely reliable, but even the most 'consistently reliable sources' are sometimes unreliable (everyone has blind spots and biases, and everyone is able to make mistakes). There are no consistent rules for which questions must be answered which way. However, if you ask these questions and find out what the answers seem to be you will have a better understanding of how reliable or unreliable a particular source is.

S.I.F.T. Method

SIFT (The Four Moves)

Use the SIFT method to separate fact from fake when reading websites and other media.

• What is its reputation?
• For deeper research, verify the information.
• Know what you're reading.
• Where is it from? Biases, point of view?
• Understand the context of the information.
• Find the best source on the subject.
• Trace claims, quotes or media back to its original context.
• Was the source you read/viewed an accurate depiction of the original?

More Information about the SIFT method , and a free 3-hour online course (five easy lessons) that will seriously improve your information evaluation skills!

!!! Do you have questions, confusions, or opinons about anything on this page, in this LibGuide, or anything else?      We are happy to Listen to and Answer Your Questions, Concerns, and more ! !!!

• << Previous: Citation Tips & Tricks
• Next: Videos about Evaluating Sources! >>
• Last Updated: Mar 13, 2024 1:29 PM
• URL: https://libguides.rtc.edu/researching_and_writing

• First Time Freshman
• Transfer Student
• Art Institute Transfer
• High School Student
• Military Student
• International Student
• Returning Student
• Current Student

Four guidelines students should follow when determining credibility of research

• November 18, 2020

*This story was updated and edited by Jenna Heil on Tuesday, October 25, 2022 for clarity and brevity.

When gathering sources for your next term paper or project, it is important to determine credibility of research. Credibility can be seen as an important aspect of establishing trustworthiness, but it does require additional research. It’s important to look at who the author is, their background and their education. You can check an author’s h-index score, which according to Bernard Becker Medical Library , is “an estimate of the importance, significance, and broad impact of cumulative research,” by using Google Scholar .

Finding reliable sources is vital for college students to succeed, so when in doubt, ask yourself if the source checks the box in the following areas:

1. Relevance Relevance is a key factor to consider when doing research. If you are studying Picasso’s blue period and come across an article about Paloma Picasso’s perfumes, ask yourself if this is relevant to your larger research project or paper. If not, it’s time to mark the resource as irrelevant and move on to different sources. If it is relevant, the source checks the first credibility requirement.

2. Authority Next, it’s time to determine the authority of the piece. Authority and credibility typically go hand in hand, but depending on the information you are looking for, authority can change. For example, a publication that focuses more on fashion design has authority in this section, but doesn’t necessarily have authority in art history. To ensure you are discovering the most authoritative academic journals, magazines or reports in your specific area of interest, we recommend utilizing RMCAD’s databases .

3. Accuracy You’re halfway there! The next credibility requirement is determining if the source is accurate. This can be tricky if you are new to a field of information. Determining accuracy requires reading and learning as much as possible about something before forming an opinion. We recommend using your best judgment when labeling something as accurate and think about what you already know when forming your final verdict of the information presented.

4. Purpose The final step is determining the purpose of the source, which requires you to learn why someone is presenting the information. A YouTube star might show creative and unique makeup techniques, but with the goal to sell makeup. A late night talk show interviewee might be selling their new movie. An opinion piece may demonstrate bias, whether it is obvious or not. These are all important things to determine.

Remember, when doing research, you want to find the best information to support your ideas, which requires careful evaluation of the information you gather. Keep in mind – relevance, authority, accuracy and purpose when determining credibility of research.

This piece was written with the help of Martha Neth from the SLC . The SLC welcomes students from all programs both on-campus and online who strive to do better in the classroom while perfecting their craft. Self-schedule an appointment with the SLC here or email the SLC at [email protected] .

Recent Blogs

• Exploring Graphic Design: A Day in the Life of a Graphic Designer
• Celebrating Heritage: Exploring the Rich Tapestry of Indigenous Art
• 15 Notable Art Festivals To Checkout in 2024
• Animating for Social Change: How Cartoons Can Drive Awareness
• RMCAD’s 60th Anniversary: Celebrating over Half a Century of Excellence in Art + Design
• Student Login

We're accepting applications for our next session. No fee, Apply Today!

Classes Starting Soon!

No Application fee

• Original article
• Open access
• Published: 31 March 2021

The role of analytical reasoning and source credibility on the evaluation of real and fake full-length news articles

• Didem Pehlivanoglu   ORCID: orcid.org/0000-0002-9082-9976 1 ,
• Tian Lin 1 ,
• Farha Deceus 1 ,
• Amber Heemskerk 1 ,
• Natalie C. Ebner 1 , 2 , 3 , 4   na1 &
• Brian S. Cahill 1   na1  

Cognitive Research: Principles and Implications volume  6 , Article number:  24 ( 2021 ) Cite this article

11k Accesses

14 Citations

7 Altmetric

Metrics details

Previous research has focused on accuracy associated with real and fake news presented in the form of news headlines only, which does not capture the rich context news is frequently encountered in real life. Additionally, while previous studies on evaluation of real and fake news have mostly focused on characteristics of the evaluator (i.e., analytical reasoning), characteristics of the news stimuli (i.e., news source credibility) and the interplay between the two have been largely ignored. To address these research gaps, this project examined the role of analytical reasoning and news source credibility on evaluation of real and fake full-length news story articles. The project considered both accuracy and perceived credibility ratings as outcome variables, thus qualifying previous work focused solely on news detection accuracy.

We conducted two independent but parallel studies, with Study 2 as a direct replication of Study 1, employing the same design but in a larger sample (Study 1: N  = 292 vs. Study 2: N  = 357). In both studies, participants viewed 12 full-length news articles (6 real, 6 fake), followed by prompts to evaluate each article’s veracity and credibility. Participants were randomly assigned to view articles with a credible or non-credible source and completed the Cognitive Reflection Test as well as short demographic questions.

Consistent across both studies, higher analytical reasoning was associated with greater fake news accuracy, while analytical reasoning was not associated with real news accuracy. In addition, in both studies, higher analytical reasoning was associated with lower perceived credibility for fake news, while analytical reasoning was not associated with perceived credibility for real news. Furthermore, lower analytical reasoning was associated with greater accuracy for real (but not fake) news from credible compared to non-credible sources, with this effect only detected in Study 2.

Conclusions

The novel results generated in this research are discussed in light of classical vs. naturalistic accounts of decision-making as well as cognitive processes underlying news articles evaluation. The results extend previous findings that analytical reasoning contributes to fake news detection to full-length news articles. Furthermore, news-related cues such as the credibility of the news source systematically affected discrimination ability between real and fake news.

Introduction

Fake news refers to “fabricated information that mimics news media content in form but not in organizational process or intent” (Lazer et al., 2018 , p. 1094). While fake news is certainly not a new occurrence—e.g., tabloid magazines have been around for nearly a century (Murray, 2013 )—its prominence in and impact on our culture has been growing. This is also related to enhanced global connectedness and broader use of online media platforms in modern society which have drastically increased access to news but also increased distribution of misinformation via fake news. One study estimated that the average American encountered between one and three fake news articles during the month prior to the 2016 presidential election (Allcott & Gentzkow, 2017 ). Given the prevalence of fake news, the relevant question is, how good are people at detecting real and fake news? Recent polls indicate that a significant portion of Americans (47%) report having difficulty distinguishing between real and fake news (Associated Press, 2019 ). Analysis of Facebook activity of the top 20 fake and real news stories showed that user engagement was greater for fake compared to real news stories (Silverman et al., 2016 ). Further, in an analysis of 126,000 real and fake news stories tweeted by about 3 million Twitter users, fake compared to real news spread more than 4.5 million times faster and in a wider range (Vosoughi et al., 2018 ).

Thus, it is crucial to investigate the processes involved in the evaluation of real and fake news. Here, we will address the following understudied research questions: (1) Is current evidence regarding an impact of analytical reasoning on fake news detection robust to methodological change (i.e., by presenting full-length articles as opposed to headlines only)?; (2) Does systematically varying the credibility of the news source influence news article evaluation?; and (3) What can we learn from examining the perceived credibility of the news articles, beyond real and fake news detection accuracy?

A cognitive account of fake and real news detection

According to Dual-Process Theory, individuals engage in two modes of information processing: a quick, intuitive mode (called System 1 ) and a slow, deliberate mode (called System 2 ; De Neys, 2012 ; Ferreira et al., 2006 ; Kahneman, 2011 ; Stanovich, 2009 ). System 1 is associated with low analytical reasoning and reliance on cognitive heuristics when making decisions (i.e., mental shortcuts based on prior knowledge and beliefs; Evans, 2007 ; Kahneman et al., 1982 ). System 2, in contrast, is associated with high analytical reasoning and involves careful and systematic consideration of information, and therefore, is less error prone than System 1.

In line with Dual-Process Theory, individuals who scored higher on a measure of analytical reasoning (i.e., Cognitive Reflection Test [CRT]; Frederick, 2005 ) were better at detecting fake news than individuals who scored low on analytical reasoning, regardless of whether the news content aligned with their political beliefs (Pennycook & Rand, 2019b ; also see Bago et al., 2020 ; Pennycook & Rand, 2019a for evidence supporting the role of analytic reasoning over and above political ideology on fake news detection). Furthermore, engagement in analytic reasoning accounted for ~ 56% to 95% of the variance in accurate detection of fake news (Pennycook & Rand, 2020 ). Lastly, while delusion-prone individuals, dogmatic individuals, and religious fundamentalists were more likely to believe fake news, these relationships were partially or fully explained by lower levels of analytical reasoning (Bronstein et al., 2019 ). In sum, there appears to be consistent evidence that lower analytical reasoning is associated with poorer fake news detection.

Current study

From previous research we know that the prevalence of fake news is significant and that individuals are poor at detecting fake news, due to low engagement of analytical reasoning. Previous research, however, focused on real and fake news detection accuracy using news headlines only, which does not capture the rich context news is frequently encountered in real life. Additionally, while previous studies considered characteristics of the evaluator (i.e., analytical reasoning), characteristics of the news stimuli (i.e., news source credibility) and the interplay between the two have been largely ignored. This paper went beyond previous work by employing full-length news articles (involving full news story along with a headline) to determine the role of: (i) analytical reasoning on evaluation of real and fake full-length news articles; (ii) credibility of the news source on evaluation of news articles; and (iii) perceived credibility of news articles, in addition to detection accuracy. Next, we will discuss the theoretical background leading to these central research aims.

Impact of analytical reasoning on real and fake news evaluation for full-length articles

In a typical fake news paradigm, participants are presented with news headlines only that are either factually accurate (real news) or not (fake news). Following each headline, participants are asked to make a set of evaluations, including, but not limited to, veracity (i.e., real vs. fake), familiarity, and willingness to share. Given that in real life, people are not typically restricted to solely using the headline to evaluate a news article (i.e., people typically can go beyond browsing headlines and read the full article), we employed full-length news articles. Limited research has attempted to shift the research field by adopting more ecologically valid news evaluation methodology. Besides being more ecologically valid, full-length articles provide rather rich contextual information and a larger set of diagnostic cues to determine credibility of the news (e.g., coherence in story line, writing and grammatical style). These additional features of full-length news articles as opposed to news headlines only inform the news evaluation process. To our knowledge only Schaewitz et al. ( 2020 ) employed full articles and found that people with high compared to those with low need for cognition were less susceptible to misinformation via fake news. Their design, however, did not involve a systematic manipulation of news veracity as they only used fake news stories. Thus, systematic variation of news veracity within a relatively more naturalistic decision-making context that allows for full exploration of the entire article, as done in the present study, has potential to further understanding of the cognitive mechanisms underlying real and fake news evaluation.

According to the Naturalistic Decision Making framework (Klein, 2008 , 2015 ), in fast-paced complex settings, decision makers mostly rely on past experiences to find the first workable option rather than trying to find the best possible solution, which requires analytical reasoning and is resource-intensive. People in real life come across real news more frequently than fake news (Allen et al., 2020 ; Guess et al., 2019 ). It is therefore possible that detection of real news relies on relatively more naturalistic decision-making processes which do not require analytical reasoning to the same extent as those involved in (less frequently encountered) fake news stories (Gigerenzer, 2007 ). Detection of fake news, in contrast, may rely more on deliberative processes that require high analytical reasoning and careful scrutinization of potential deceptive cues; which full-length news articles may be more diagnostic of than (brief) headlines. Based on these considerations, we predicted that higher analytical reasoning would be associated with increased fake news accuracy, while there would be no relationship between analytical reasoning ability and real news detection accuracy (Hypothesis 1).

Effects of systematic variation of news source credibility on real and fake news evaluation

The Elaboration Likelihood Model put forth by Petty and Cacioppo ( 1986 ) is a dual-process model of persuasion. According to this model, information is processed via a central, systematic route when the decision maker is both motivated and has the necessary cognitive resources to do so. However, when the decision maker lacks either the necessary motivation or the cognitive resources, they will process information via a peripheral, heuristic route. Importantly, this model posits that heuristic cues such as the credibility of the source (in our case the news source of the article) will have a greater effect when the decision maker is processing the message via the peripheral route (Carpenter, 2015 ; Petty & Cacioppo, 1986 ; Ratneshwar & Chaiken, 1991 ). Thus, it is possible that news source credibility moderates real and fake news evaluation, especially when information is processed peripherally (i.e., involving lower analytical reasoning).

To our knowledge, there are no studies examining the impact of analytical reasoning on accuracy for both real and fake news under systematic variation of news source credibility. Given that individuals rely more on heuristics as cognitive resources decrease (Cacioppo et al., 1986 ; Petty & Cacioppo, 1986 ) and that low analytical reasoning is associated with reduced ability to detect fake news (Bronstein et al., 2019 ; Pennycook & Rand, 2019b ), we hypothesized that lower analytical reasoning would be associated with increased accuracy for real and particularly fake news paired with a credible compared to a non-credible news source ( Hypothesis 2 ).

Beyond accuracy, the role of perceived credibility on real and fake news evaluation

Most fake news studies have focused on accuracy as the primary outcome measure, while neglecting perceived credibility of real and fake news as relevant evaluation metric. Pennycook and Rand ( 2019a ) demonstrated that mainstream online news sources (e.g., cnn.com; npr.org) were perceived as more credible than online sources of partisan (e.g., breitbart.com; dailykos.com) or fake (e.g., thelastlineofdefense.org; now8news.com) news. This finding suggests that the source of a news item may be an important piece of information when evaluating the credibility of an article. Indeed, Luo et al. ( 2020 ) showed that perceived credibility of news headlines was greater when paired with more credible news sources (but see Schaewitz et al., 2020 for no effect of news source on perceived credibility of fake news).

Based on this evidence, we propose that perceived credibility may constitute a relevant, but currently understudied, construct involved in news evaluation. We hypothesized that higher analytical reasoning would be associated with less perceived credibility for fake news, while analytical reasoning ability would not affect perceived credibility of real news ( Hypothesis 3 ). Furthermore, we predicted that lower analytical reasoning would be associated with greater perceived credibility for real and particularly fake news paired with a credible compared to a non-credible news source ( Hypothesis 4 ).

To enhance scientific rigor and reproducibility (Open Science Collaboration, 2015 ), we adopted a two-study approach in this paper. In particular, we conducted two parallel but independent studies to systematically test in Study 1 and replicate with a large sample in Study 2 our research hypotheses.

Participants

Study 1 recruited 360 undergraduates from the Department of Psychology’s SONA system. A total of 68 participants were removed from the final analysis for the following reasons: 3 had average reading times 3 SD s greater than the group average, 41 had incomplete news evaluation data, and 24 failed the attention checks (e.g., Please answer 2 to this question ). The final analysis sample in Study 1 thus comprised 292 participants.

Study 2 used the same recruitment methods as Study 1; assuring through the SONA system that not the same participants were enrolled across the two studies. The initial sample consisted of 424 undergraduate students. A total of 67 participants were removed from the final analysis for the following reasons: 1 had average reading times 3 SD s greater than the group average, 42 had incomplete news evaluation data, and 24 failed the attention checks. The final analysis sample for Study 2 thus comprised 357 participants. Table 1 presents sample characteristics for participants in Study 1 and Study 2.

Both studies adopted a 2 (Veracity: real vs. fake; dichotomous; within-subjects) × 2 (Source: credible vs. non-credible; dichotomous; between-subjects) mixed design. Participants were randomly assigned to evaluate 6 real and 6 fake news articles either from credible ( N  = 138 in Study 1; N  = 171 in Study 2) or non-credible ( N  = 154 in Study 1; N  = 186 in Study 2) news sources (see below for more details).

Study materials were identical in Study 1 and 2.

News articles

To select fake news articles, we used the “junk news” archive maintained by the reputable fact-checking website Snopes.com (Junk News Archives, n.d.). For real news articles, we used the “true news” archive maintained by Snopes ( www.snopes.com/archive/ ) which involves news articles from reputable news organizations (e.g., Washington Post, NPR). From these archives, we selected 6 fake and 6 real news articles that varied by topic, including healthcare (e.g., doctors refusing care on religious grounds), religion (e.g., Mormonism and same-sex marriage, Pope Francis), education (e.g., California textbooks, guns on campuses), crime (e.g., prison escape, felony assault), and politics (e.g., the Black Lives Matter movement, gun confiscations). We conducted an independent pilot study with 98 college students from the Department of Psychology’s SONA system to assess the credibility of the selected 12 news articles (i.e., How credible was this news article? ; rated on a scale from 1 =  Not at all credible to 10 =  Completely credible ). Real news articles were rated as more credible ( M  = 5.90, SD  = 1.09) than fake news articles ( M  = 4.00, SD  = 1.39); t (97) = 13.40, p  < 0.001).

We conducted an additional independent pilot study with 161 college students from the Department of Psychology’s SONA system to determine the final set of news sources for use in our study paradigm. Participants were asked to indicate the level of credibility ( How credible is this news source? ) on a scale from 1 =  Not at all credible to 10 =  Completely credible for 10 commonly known news organizations (i.e., 5 credible sources: NPR, CNN, Washington Post, New York Times, BBC; 5 non-credible sources: True Pundit, Conservative Daily News, World News Daily Report, Liberty Writers News, Red State). The three sources with the highest averages (i.e., NY Times [ M  = 7.00, SD  = 2.30], Washington Post [ M  = 6.84, SD  = 2.23], and NPR [ M  = 6.80, SD  = 2.21]) were selected as “credible sources” and the three sources with the lowest averages (i.e., True Pundit [ M  = 4.30, SD  = 1.70], Red State [ M  = 4.34, SD  = 1.73], and Conservative Daily News [ M  = 4.55, SD  = 1.83]) were selected as “non-credible sources” for use in the study. Additional file 1 : Appendix A provides a full set of the news articles used in this project.

We created two experimental lists to control pairing of Veracity of the news article (real vs. fake; within-subjects) and Credibility of the news source (credible vs. non-credible; between-subjects). The two lists comprised the same 12 unique articles and were counterbalanced across participants. In List 1, the 6 real and the 6 fakes news articles were randomly paired with credible news sources (i.e., NY Times, Washington Post, NPR; credible condition). In List 2, the 6 real and the 6 fakes news articles were randomly paired with non-credible news sources (i.e., True Pundit, Red State, Conservative Daily News; non-credible condition). Presentation order within each list was pseudorandomized, with the constraint that the same type of news articles (real vs. fake) was not repeated more than two times in a row. For each list, (approximately) half of the participants received the reversed order to counter order effects.

• Cognitive reflection test

The CRT (Frederick, 2005 ) is a three-item task designed to measure the degree to which analytical reasoning is used when solving problems. For example, one item asks: “ A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? ” Participants high in analytical reasoning would overcome the impulse to give the intuitive answer 10 cents and would instead give the correct answer of 5 cents. Thus, a greater CRT score reflects higher analytical reasoning.

Study procedures for Study 1 and Study 2 were identical unless noted otherwise. Participants accessed the study link through the SONA system website and completed the study remotely through Qualtrics ( https://www.qualtrics.com/ ). Prior to study enrollment, all participants consented electronically to participate.

During the News Evaluation Task , participants were presented with 12 news articles (6 real, 6 fake). Each article was presented on the screen for at least 60 s to ensure sufficient reading time, as determined in an internal pilot. Beyond the 60-s window, the task was self-paced. Footnote 1 After reading each article, participants were prompted with the following questions (in this order): accuracy ( Is this news article real or fake? ; response option: Real vs. Fake ), confidence ( How confident are you in your decision regarding the authenticity of this news article? ; response option: 1 ( Not at all confident ) to 10 ( Completely confident )), perceived credibility ( How credible do you find this news article? ; response option: 1 ( Not at all credible ) to 10 ( Completely credible )), media sharing ( Would you share this news article on social media? ; response option: Yes vs. No ), and familiarity ( Have you seen this article before? ; response option: Yes vs. No ). Participants were not informed about the number of articles presented to them to avoid response biases (e.g., 50/50 real vs. fake response rate).

After evaluating the news articles, participants completed the CRT and a short demographic questionnaire. Footnote 2 Study duration was about 1 h in each of the two studies.

Data analysis

We used multilevel random intercept models (Gelman & Hill, 2007 ; Hox, 2010 ) to accommodate for the nested data structure. Specifically, we conducted cross-random effects analyses with cross-classification of news articles and participants, and a nesting structure for repeated observations within participants. This approach allows evaluations made by the same participant to be correlated across different news articles, as well as accounts for dependencies of evaluations of the same news article made by different participants.

Our analyses included two separate models, one for accuracy Footnote 3 and one for perceived credibility. Complete datasets and analysis code can be found at https://osf.io/yrabp/ . For the binary outcome variable accuracy (0 = wrong, 1 = correct), we used mixed effects logistic regression; for the ordinal/continuous outcome variables perceived credibility we employed multilevel regression. Each model considered the fixed effect of veracity of the news article (0 = real, 1 = fake), credibility of the source (0 = credible, 1 = non-credible), and the CRT score of each participant (continuous variable) as predictors. We further estimated the interactions between these independent variables in each model. We also entered the random intercepts of evaluations for news articles and participants to estimate the variability of mean evaluations across news articles and participants, respectively. Reading time (beyond the fixed 60-s window), familiarity, gender, and presentation order were entered as covariates.

We applied maximum likelihood estimation for all model parameters and used the Wald tests to determine significance of the effects. For significant interactions, we compared (using z tests for pairwise comparisons) and plotted predicted marginal means (using a mean of 0 and ± 1 SD for interactions involving the continuous CRT variable) from the estimated model parameters to facilitate understanding of significant interactions. All analyses were performed in Stata 16.1 (StataCorp, 2019 ).

Consistent across both studies, the Veracity × CRT interaction on accuracy was significant [Study 1: χ 2 (1)  = 23.84, p  < 0.001; Study 2: χ 2 (1)  = 10.78, p  = 0.001]. As shown in Fig.  1 , real news accuracy did not change across levels of analytical reasoning (indexed by CRT scores) [Study 1/Panel A: z  = 1.37, p  = 0.339; Study 2/Panel B: z  = 0.5, p  = 0.619]. Accuracy for fake news, however, increased with higher analytical reasoning [Study 1/Panel A: z  = 4.53, p  < 0.001; Study 2/Panel B: z  = 4.13, p  < 0.001], thus supporting Hypothesis 1 . Furthermore, also consistent across both studies and depicted in Fig.  1 , higher analytical reasoning was associated with better detection of fake than real news [Study 1/Panel 1A: z  = 3.79, p  < 0.001; Study 2/Panel 1B: z  = 3.43, p  = 0.001].

Percent accuracy for real (gray line) and fake (black line) news articles across levels of analytical reasoning (continuous; indexed by Cognitive Reflection Test (CRT) scores) in Study 1 (Panel A) and Study 2 (Panel B). Error bars denote standard errors. The medium analytical reasoning level indicates the mean CRT score in the current sample while the low and high levels indicate 1 SD below and above the mean CRT score, respectively. The y-axis start point reflects the 50% chance level. Consistent across both studies, real news accuracy did not change across levels of analytical reasoning, while accuracy for fake news increased with higher analytical reasoning

The three-way interaction between Veracity × CRT × Source was not significant in Study 1 ( χ 2 (1)  = 0.42, p  = 0.517), but was significant in Study 2 ( χ 2 (1)  = 6.2, p  = 0.013). In particular, as shown in Fig.  2 (for Study 2), lower analytical reasoning was associated with greater accuracy for real news from credible compared to non-credible sources ( z  = 3.42, p  = 0.001). Furthermore as depicted in Fig.  2 , news source credibility did not influence accuracy for fake news across levels of analytical reasoning ( z s < 1, p s > 0.272); and higher analytical reasoning was associated with greater accuracy for fake news irrespective of news source credibility (all z s > 2.55, p s < 0.02). These findings partially supported Hypothesis 2 . Footnote 4

Veracity × CRT × Source interaction in Study 2; this 3-way interaction was not significant in Study 1. Percent accuracy for real (gray lines) and fake (black lines) news articles from credible (solid lines) and non-credible (dashed lines) news sources across levels of analytical reasoning (continuous; indexed by Cognitive Reflection Test (CRT) scores) in Study 2. Error bars denote standard errors. The medium analytical reasoning level indicates the mean CRT score in the current sample while the low and high levels indicate 1 SD below and above the mean CRT score, respectively. The y-axis start point reflects the 50% chance level. Lower analytical reasoning was associated with greater accuracy for real news paired with credible compared to non-credible sources, while news source did not influence accuracy for fake news across levels of analytical reasoning

• Perceived credibility

Consistent across both studies, the Veracity × CRT interaction was significant [Study 1: χ 2 (1)  = 14.28, p  < 0.001; Study 2: χ 2 (1)  = 24.57, p  < 0.001]. As depicted in Fig.  3 , perceived credibility for real news was overall higher than perceived credibility for fake news and was not influenced by levels of analytical reasoning [Study 1/Panel A: z  = 0.97, p  = 0.66; Study 2/Panel B: z  = 0.52, p  = 0.6]. In contrast, higher analytical reasoning was associated with less perceived credibility for fake news [Study 1/Panel A: z  = 4.22, p  < 0.001; Study 2/Panel B: z  = 3.55, p  < 0.001], in line with Hypothesis 3 .

Mean perceived credibility rating (1 =  Not at all credible to 10 =  Completely credible ) for real (gray line) and fake (black line) news articles across levels of analytical reasoning (continuous; indexed by Cognitive Reflection Test (CRT) scores) in Study 1 (Panel A) and Study 2 (Panel B). Error bars denote standard errors. The medium analytical reasoning level indicates the mean CRT score in the current sample while the low and high levels indicate 1 SD below and above the mean CRT score, respectively. Note that the y-axis ranges from 1 to 7 to reflect the actual range of responses given by participants. Consistent across both studies, perceived credibility for real news was not influenced by levels of analytical reasoning, while higher analytical reasoning was associated with less perceived credibility for fake news

The three-way interaction between Veracity × CRT × Source was not significant in either of the studies [Study 1: χ 2 (1)  = 1.49, p  = 0.222; Study 2: χ 2 (1)  = 0.67, p  = 0.413]. Thus, our data did not support Hypothesis 4 .

The present two-study project, with a built-in replication, is the first to examine evaluation of both real and fake news under consideration of cognitive factors (i.e., analytical reasoning), characteristics of the news stimuli (i.e., source credibility) as well as the interplay between the two using a novel, relatively more ecologically valid full-length article paradigm. In addition, our approach went beyond investigation of real and fake news evaluation accuracy in also determining effects on the perceived credibility of the articles. Consistent across both studies, higher analytical reasoning was associated with greater accuracy and reduced perceived credibility for fake news, while analytical reasoning ability did not moderate accuracy and perceived credibility of real news. Furthermore, in Study 2 (but not in Study 1) news source credibility influenced the relationship between analytical reasoning ability and news detection accuracy for real (but not fake) news. These novel findings have potential to advance theory and empirical understanding of cognitive processes underlying news evaluations, as discussed next.

Higher analytical reasoning improves fake news detection in full-length articles

Consistently across both studies and in line with our predictions, higher analytical reasoning was associated with more accurate detection of fake news articles. Thus, extending previous evidence from headlines-only studies (Bronstein et al., 2019 ; Pennycook & Rand, 2019a , 2020 ; Pennycook et al., 2015 ), by using full-length news articles the present study provides support for a role of analytical reasoning on fake news detection. In line with our prediction, real news accuracy, in contrast, was not influenced by analytical reasoning ability. As real news is more common in everyday life than fake news, detection of real news may not be as resource-demanding than detection of fake news, possibly underlying the moderating effect of analytical reasoning on fake but not real news detection. The Naturalistic Decision Making framework (Klein, 2008 , 2015 ) highlights the role of relatively automatic (intuitive) and experience-based successful decision making in naturalistic real-world settings. This framework may be particularly fruitful in future research on determining the mechanisms underlying news evaluation. As touched on earlier, we believe that our full-length article approach is more representative of how news articles are typically encountered in real life (e.g., with rich contextual information), thus allowing to better capture complex cognitive processes involved in naturalistic news evaluation. To further improve ecological validity, future research could leverage real or simulated social media platforms (e.g., Twitter, Facebook), where people directly interact with the news (see Lin et al., 2019 , for a similar approach in email phishing detection). This approach would also be in line with research demonstrating the importance of using ecological valid task formats to improve performance (Evans, 2011 ; Mercier & Sperber, 2011 ). The present study constitutes a first important step in this direction.

Further, consistent across both studies, higher analytical reasoning was associated with better detection of fake than real news. One could argue that better detection of fake compared to real news with higher analytical reasoning may simply reflect a response bias (i.e., tendency to overclaim news as fake, which could be an artifact of task instructions). However, results from an additional analysis we conducted that controlled for sensitivity and response bias did not support this interpretation. Instead, this finding may reflect an enhanced ability to detect deceptive cues inherent in fake news stories among individuals who engage in higher levels of analytical reasoning. That is, diagnostic cues and details in the full-length fake news articles used in this study such as pertaining to formatting, grammar issues, general writing style (and that may not be present in real news articles) may have facilitated fake news detection among individuals who engage in deeper processing (i.e., higher analytical reasoning). These explanations are rather speculative and warrant research that uses natural language processing machine learning approaches (Gilda, 2017 ; Oshikawa et al., 2018 ), for example, to determine deception-related diagnostic cues in fake (relative to real) news and to further clarify the interplay between these cues and analytical reasoning ability in news detection.

Lower analytical reasoning enhances detection of real news paired with credible sources

We found that lower analytical reasoning was associated with better detection of real news paired with credible sources, while news source credibility did not influence accuracy for fake news across levels of analytical reasoning. To date only a small number of studies have examined the impact of source credibility on news detection accuracy. Luo et al. ( 2020 ) showed that reliability of the source (indexed by a high number of Facebook likes) increased the detection of real news but decreased the detection of fake news. In contrast, Schaewitz et al. ( 2020 ) found no effect of source credibility (i.e., fictitious news sources that were rated on credibility) on fake news accuracy. Furthermore, Pennycook and Rand ( 2020 ) reported a negative association between analytical reasoning and susceptibility to fake news, regardless of whether a news source was present or absent, suggesting no moderating effect of source credibility on the relationship between analytical reasoning and fake news detection (also see Dias et al., 2020 for similar results).

Our study contributes to this literature and is the first to suggest that news source credibility may influence news detection as a function of analytical reasoning in full-length real (but not fake) news articles. However, this finding only emerged in Study 2 but not in Study 1 and thus needs to be interpreted with caution. It is possible that lower analytical reasoning reflects greater reliance on source heuristics. In fact, our results are consistent with predictions from the Elaboration Likelihood Model (Petty & Cacioppo, 1986 ), which proposes that peripheral cues such as the credibility of the source of a message, more likely influence individuals low in cognitive resources as they engage in less elaborative or systematic processing; a possible explanation that can be systematically explored in future work. Also, as the three-way Veracity × CRT × Source interaction was only significant in Study 2, which comprised a larger sample size, but not in Study 1, a future replication of this effect in a sample of at least the size as in Study 2 is warranted to corroborate the finding. Additionally, because of study duration related constraints and our preference for keeping our news article material uniform across participants (i.e., each participant viewed the same real and fake news articles), the credibility of the source the news articles were paired with was manipulated between participants in this project. This design feature may have reduced statistical power to detect significant effects related to source credibility (i.e., one would expect greater sensitivity of a factor that is manipulated within-subjects (in this case, veracity) than one that is manipulated between-subjects (in this case, news source credibility)). Future studies could employ a within-subjects design to investigate this possibility.

Beyond accuracy, perceived credibility as an additional route to study cognitive mechanisms underlying news evaluation

Overall, perceived credibility for real news was higher than perceived credibility for fake news in both studies. Furthermore, and again consistent across both studies, higher analytical reasoning was associated with lower perceived credibility for fake news, while perceived credibility for real news did not vary by level of analytical reasoning.

Somewhat in contrast to our findings pertaining to accuracy, news source did not moderate the effect of analytical reasoning on perceived credibility of real vs. fake news. Specifically, participants who relied more on analytical reasoning were better at detecting fake news and rated fake news as less credible. Importantly, the credibility of the news source did not affect accuracy or perceived credibility of fake news in individuals high on analytical reasoning. This finding may suggest that individuals high on analytical reasoning utilize diagnostic cues and contextual features provided within the fake news article itself (e.g., sentiment, formatting style, grammar issues, general writing style).

If this interpretation is true, then this highlights two important implications for future research. First, future research may benefit from using full-length news articles because headlines only contain a finite amount of diagnostic cues and may strip away important information to discern between real and fake news. Given that our current results (using full-length articles) align with past research that used only headlines, future research needs to directly compare full-length articles with headlines only and by systematically manipulating news source among individuals with varying levels of analytical reasoning to better assess these claims. Second, the aforementioned pattern emerged clearer by collecting novel outcome measures (i.e., perceived credibility of the news), thus, supporting the need for future research to explore other (sensitive) outcome measures (e.g., news content related questions) that may help gain a more complete understanding of the phenomenological process individuals engage in when detecting fake news.

Additionally, the possibility that participants may have directed their attention primarily towards the news stories and its central content (e.g., sentiment, language style) rather than peripheral cues (e.g., the news source) can be further investigated using eye-tracking. This technique will allow determination of eye gaze patterns as well as physiological reactions associated with arousal levels (e.g., pupil dilation) when interacting with news stories. These innovative methodological approaches would not only help identifying candidate cognitive mechanisms but could also inform targeted interventions (e.g., eye-tracking guided reading intervention to train people to process information relevant to detection of deceptive cues). This rich data will also lend itself particularly well to computational modeling approaches to describe decision-making processes underlying deception detection (see Hakim et al., 2020 for a computational modeling approach to phishing email detection).

Future research directions

Our study, like the majority of previous work, focused on a rather homogeneous (e.g., in terms of race/ethnicity and age) sample. Based on growing evidence that sensitivity for detection of deceptive cues decreases with chronological age (Ebner et al., 2020 ; Grilli et al., in press; Zebrowitz et al., 2018 ) as well as varies by gender and marital status (Alves & Wilson, 2008 ), education (Wood et al., 2018 ), and income (James et al., 2014 ), we propose examining fake news detection using more diverse samples to move this research forward (Pehlivanoglu et al., 2020 ). For example, older compared to younger individuals were more likely to share fake news (Grinberg et al., 2019 ; Guess et al., 2019 ). A recent narrative review by Brashier and Schacter ( 2020 ) argues that susceptibility to fake news with age may not only depend on cognitive decline, but may also be related to age-related changes in socioemotional functioning (e.g., increase in positive emotion and interpersonal trust) as well as in expertise with online news media platforms. Thus, examining the role of expertise with online news media outlets (e.g., indexed by digital literacy; Sengpiel & Dittberner, 2008 , and news media literacy; Maksl et al., 2015 ) on the relationship between analytical reasoning and real vs. fake news evaluation in a sample of adults varying in age (college students vs. middle-aged adults vs. older adults) is a fruitful future research direction. These future age-comparative studies would also be helpful to identify mechanisms that may render certain groups particularly vulnerable to fake news and would open tremendous potential for interventional approaches, including particular at-risk populations (Ebner, et al., in press).

Future studies should also set out to determine the specific dynamics of the impact of analytical reasoning on real and fake news evaluation. For example, it is possible that news related variables such as news topics/content (e.g., politics vs. pop culture) differentially call on analytical reasoning ability when evaluating real and fake news articles. In addition, it is possible that individuals can flexibly allocate their resources and switch between processing modes (e.g., effortful vs. non-effortful thinking; shallow vs. deep processing) for improved news evaluation. Utilizing neuroimaging techniques (e.g., fMRI) could help outline the neurocognitive mechanisms underlying news evaluation. Event-related potentials could help determine temporal dynamics of engagement in different levels of reasoning during news evaluation (e.g., whether engagement in analytic reasoning changes during early vs. late stages of processing; whether one reasoning mode is replaced by the other over time; whether news-related variables such as source credibility moderates these processes).

This study is the first to demonstrate a positive association between analytical reasoning and fake news detection accuracy using full-length news articles, as a relatively more ecologically valid approach in research on news evaluation. The study is also first in supporting a moderating role of news source credibility in the endeavor to delineate cognitive mechanisms underlying news evaluation; and it advances knowledge pertaining to perceived credibility of news as an alternative outcome variable to accuracy. Across two independent studies, findings from this research underline the importance of both individual differences and news-related characteristics when evaluating news. Our research has potential for theoretical advancement regarding relative contributions of rational vs. more naturalistic decision making in the applied context of fake news detection. Employing full-length news articles, novel findings reported here spur future research hypotheses regarding the (neuro)cognitive mechanisms involved in detection of deceptive cues in news evaluation as well as possible intervention designs to tackle the major and daily growing threat of misinformation from fake news, at both individual and societal levels.

Availability of data and materials

The stimulus set, complete datasets used in the analyses, and analysis code are available in the OSF repository, https://osf.io/yrabp/ . None of the experiments were preregistered.

Reading time data (in seconds; averaged across real and fake news articles) showed that participants took more than 60 s on average [Mean = 106.15 (Study 1), 101.53 (Study 2); Median = 89.25 (Study 1), 87.26 (Study 2); SD  = 48.69 (Study 2), 48.13 (Study 2); Range = 62.15–363.15 (Study 1), 61.93–404.48 (Study 2)], suggesting that the news articles were processed adequately.

Both Study 1 and Study 2 also included the Gullibility Scale (Teunisse et al., 2020 ) and the short form of the Need for Cognition Scale (Cacioppo et al., 1984 ). Study 2 additionally included measures for religiosity (Batson & Schoenrade, 1991 ), spirituality (Büssing et al., 2007 ), conservatism (Everett, 2013 ), and media consumption habits (adopted from Maksl et al., 2015 ). These additional constructs were outside the scope of this report and were therefore not included in the statistical analysis for parsimony.

To support findings for accuracy, we conducted parallel analyses on confidence ratings and report the results in Additional file 2 : Appendix B. News sharing was not analysed as an outcome measure due to floor effects in “yes” responses (see Table B1 in Additional file 2 : Appendix B).

To ensure that our results regarding accuracy were not confounded by response bias, based on signal detection theory (Macmillan & Creelman, 2004 ), we computed sensitivity (d’ = z(Hit rate)—z(False alarm rate)) and response bias (c = − 0.5[z(Hit rate) + z(False alarm rate)]) for each participant, in both Study 1 and Study 2. Then, we added the scores for sensitivity and response bias as covariates and re-ran the analyses pertaining to accuracy. This re-analysis resulted in the same findings as our original analysis. In particular, the Veracity × CRT interaction was significant in both studies [Study 1: χ 2 (1)  = 21.92, p  < 0.001; Study 2: χ 2 (1) ) = 10.01 p  = 0.002]. The three-way interaction between Veracity × CRT × Source was not significant in Study 1 ( χ 2 (1)  = 0.03, p  = 0.857), but was significant in Study 2 ( χ 2 (1)  = 5.8, p  = 0.016).

Abbreviations

Allcott, H., & Gentzkow, M. (2017). Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31 (2), 211–236.

Article   Google Scholar  

Allen, J., Howland, B., Mobius, M., Rothschild, D., & Watts, D. J. (2020). Evaluating the fake news problem at the scale of the information ecosystem. Science Advances , 6 (14).

Alves, L., & Wilson, S. (2008). The effects of loneliness on telemarketing fraud vulnerability among older adults. Journal of Elder Abuse & Neglect, 20 (1), 63–85.

Bago, B., Rand, D. G., & Pennycook, G. (2020). Fake news, fast and slow: deliberation reduces belief in false (but not true) news headlines. Journal of Experimental Psychology: General , 1–18.

Batson, C. D., & Schoenrade, P. A. (1991). Measuring religion as quest: 2) reliability concerns. Journal for the Scientific Study of Religion, 30 (4), 430–447.

Brashier, N. M., & Schacter, D. L. (2020). Aging in an era of fake news. Current Directions in Psychological Science, 29 (3), 316–323.

Article   PubMed   Google Scholar  

Bronstein, M. V., Pennycook, G., Bear, A., Rand, D. G., & Cannon, T. D. (2019). Belief in fake news is associated with delusionality, dogmatism, religious fundamentalism, and reduced analytic thinking. Journal of Applied Research in Memory and Cognition, 8 (1), 108–117.

Büssing, A., Ostermann, T., & Matthiessen, P. F. (2007). Distinct Expressions of Vital Spirituality "The ASP Questionnaire as an Explorative Research Tool". Journal of Religion and Health , 267–286.

Cacioppo, J. T., Petty, R. E., Chuan, F. K., & Rodriguez, R. (1986). Central and peripheral routes to persuasion. An individual difference perspective. Journal of Personality and Social Psychology, 51 (5), 1032–1043.

Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48 (3), 306–307.

Carpenter, C. J. (2015). A meta-analysis of the Elm’s argument quality × processing type predictions. Human Communication Research, 41 (4), 501–534.

De Neys, W. (2012). Bias and conflict: A case for logical intuitions. Perspectives on Psychological Science, 7 (1), 28–38.

Dias, N., Pennycook, G., & Rand, D. G. (2020). Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media. Harvard Kennedy School Misinformation Review , 1 (1).

Ebner, N. C., Ellis, D. M., Lin, T., Rocha, H. A., Yang, H., Dommaraju, S., Soliman, A., Woodard, D. L., Turner, G. R., Spreng, R. N., & Oliveira, D. S. (2020). Uncovering susceptibility risk to online deception in aging. The Journals of Gerontology: Series B Psychological Sciences and Social Sciences, 75 (3), 522–533.

Ebner, N. C., Pehlivanoglu, D., Polk, R., Turner, G. R., & Spreng, R. N. (in press). Aging online: Rethinking the aging decision maker in a digital era . In Y. Hanoch & S. Wood (Eds). A fresh look at fraud: theoretical and applied approaches (behavioral economics and healthy behaviors). (Routledge Taylor Francis).

Evans, J. S. B. T. (2007). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59 (1), 255–278.

Evans, J. S. B. T. (2011). Reasoning is for thinking, not just for arguing. Behavioral and Brain Sciences, 34 (2), 77–78.

Everett, J. A. C. (2013). The 12 item social and economic conservatism scale (SECS). PLoS ONE, 8 (12), e82131.

Article   PubMed   PubMed Central   Google Scholar  

Ferreira, M. B., Garcia-Marques, L., Sherman, S. J., & Sherman, J. W. (2006). Automatic and controlled components of judgment and decision making. Journal of Personality and Social Psychology .

Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19 (4), 25–42.

Gelman, A., & Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models . . Cambridge University Press.

Google Scholar  

Gigerenzer, G. (2007). Gut feelings: The intelligence of the unconscious . . Penguin.

Gilda, S. (2017). Notice of violation of IEEE publication principles: Evaluating machine learning algorithms for fake news detection. In 2017 IEEE 15th student conference on research and development (SCOReD) , (pp. 110–115).

Grilli, M. D., McVeigh, K. S., Hakim, Z. M., Wank, A. A., Getz, S. J., Levin, B. E., Ebner, N. C., & Wilson, R. C. (in press). Is this phishing? Older age is associated with greater difficulty discriminating between safe and malicious emails. Journal of Gerontology: Psychological Sciences.

Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., & Lazer, D. (2019). Fake news on Twitter during the 2016 US presidential election. Science, 363 (6425), 374–378.

Guess, A., Nagler, J., & Tucker, J. (2019). Less than you think: Prevalence and predictors of fake news dissemination on Facebook. Asian-Australasian Journal of Animal Sciences, 32 (2), 1–9.

Hakim, Z. M., Ebner, N. C., Oliveira, D. S., Getz, S. J., Levin, B. E., Lin, T., Grilli, M. D., & Wilson, R. C. (2020). The Phishing Email Suspicion Test (PEST) a lab-based task for evaluating the cognitive mechanisms of phishing detection. Behavior Research Methods, 1–11.

Hox, J. J. (2010). Multilevel analysis: Techniques and applications . . Routledge.

Book   Google Scholar  

James, B. D., Boyle, P. A., & Bennett, D. A. (2014). Correlates of susceptibility to scams in older adults without dementia. Journal of Elder Abuse & Neglect, 26 (2), 107–122.

Kahneman, D. (2011). Thinking, fast and slow . . Macmillan. https://doi.org/10.2307/1914185 .

Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases . . Cambridge University Press.

Klein, G. (2008). Naturalistic decision making. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50 (3), 456–460.

Klein, G. (2015). A naturalistic decision making perspective on studying intuitive decision making. Journal of Applied Research in Memory and Cognition, 4 (3), 164–168.

Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., & Schudson, M. (2018). The science of fake news. Science, 359 (6380), 1094–1096.

Lin, T., Capecci, D. E., Ellis, D. M., Rocha, H. A., Dommaraju, S., Oliveira, D. S., & Ebner, N. C. (2019). Susceptibility to spear-phishing emails: Effects of internet user demographics and email content. ACM Transactions on Computer-Human Interaction, 26 (5), 1–28.

Luo, M., Hancock, J. T., & Markowitz, D. M. (2020). Credibility perceptions and detection accuracy of fake news headlines on social media: Effects of truth-bias and endorsement cues. Communication Research , 0093650220921321.

Macmillan, N. A., & Creelman, C. D. (2004). Detection theory: A user’s guide . . Psychology Press.

Maksl, A., Ashley, S., & Craft, S. (2015). Measuring news media literacy. Journal of Media Literacy Education, 6 (3), 29–45.

Mercier, H., & Sperber, D. (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34 (2), 57–74.

Murray, L. (2020). National enquirer. Encyclopedia Britannica, 17 Dec. 2013, https://www.britannica.com/topic/National-Enquirer . Retrived July 20, 2020.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716.

Oshikawa, R., Qian, J., & Wang, W. Y. (2018). A survey on natural language processing for fake news detection. https://arxiv.org/abs/1811.00770 .

Pehlivanoglu, D., Lin, T., Chi, K., Perez, E., Polk, R., Cahill, B., Lighthall, N., Ebner, N. C. (2020). News veracity detection among older adults during the COVID-19 pandemic: The role of analytical reasoning, mood, news consumption, and news content. https://doi.org/10.31234/osf.io/3kgq9

Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). Everyday consequences of analytic thinking. Current Directions in Psychological Science, 24 (6), 425–432.

Pennycook, G., & Rand, D. G. (2019a). Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences of the United States of America, 116 (7), 2521–2526.

Pennycook, G., & Rand, D. G. (2019b). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition, 188 , 39–50.

Pennycook, G., & Rand, D. G. (2020). Who falls for fake news? The roles of bullshit receptivity, overclaiming, familiarity, and analytic thinking. Journal of Personality, 88 (2), 185–200.

Petty, R. E., & Cacioppo, J. T. (1986). The elaboration likelihood model of persuasion. In Communication and persuasion: Central and peripheral routes to attitude change (pp. 1–24).

Associated Press. (2019). AP-NORC/USAFacts poll: Americans struggle to ID true facts. AP News. Retrieved from https://apnews.com/article/c762f01370ee4bbe8bbd20f5ddf2adbe

Ratneshwar, S., & Chaiken, S. (1991). Comprehension’s role in persuasion: The case of its moderating effect on the persuasive impact of source cues. Journal of Consumer Research, 18 (1), 52–62.

Schaewitz, L., Kluck, J. P., Klösters, L., & Krämer, N. C. (2020). When is disinformation (in)credible? Experimental findings on message characteristics and individual differences. Mass Communication and Society, 23 (4), 484–509.

Sengpiel, M., & Dittberner, D. (2008). The computer literacy scale (CLS) for older adults-development and validation. Mensch und Computer 2008-Tagungsband.

Silverman, C., Strapagiel, L., Shaban, H., Hall, E., & Singer-Vine, J. (2016). Hyperpartisan Facebook pages are publishing false and misleading information at an alarming rate. Buzzfeed News, 20 , 68.

Stanovich, K. E. (2009). Distinguishing the reflective, algorithmic, and autonomous minds: Is it time for a tri-process theory? In In two minds: Dual processes and beyond. (pp. 55–88).

StataCorp. . (2019). Stata statistical software: Release 16 . . StataCorp LP.

Teunisse, A. K., Case, T. I., Fitness, J., & Sweller, N. (2020). I Should have known better: Development of a self-report measure of gullibility. Personality and Social Psychology Bulletin, 46 (3), 408–423.

Vosoughi, S., Roy, D., & Aral, S. (2018). News on-line. Science, 1151 , 1146–1151.

Wood, S., Liu, P.-J.J., Hanoch, Y., Xi, P. M., & Klapatch, L. (2018). Call to claim your prize: Perceived benefits and risk drive intention to comply in a mass marketing scam. Journal of Experimental Psychology: Applied, 24 (2), 196–206.

PubMed   Google Scholar  

Zebrowitz, L. A., Ward, N., Boshyan, J., Gutchess, A., & Hadjikhani, N. (2018). Older adults’ neural activation in the reward circuit is sensitive to face trustworthiness. Cognitive, Affective and Behavioral Neuroscience, 18 (1), 21–34.

Download references

Acknowledgements

We appreciate the willingness of our participants to contribute to this research.

This research was funded by the Department of Psychology, College of Liberal Arts and Science, University of Florida and NIH/NIA Grant 1R01AG057764.

Author information

Senior authorship: Natalie C. Ebner and Brian S. Cahill

Authors and Affiliations

Department of Psychology, University of Florida, 945 Center Dr, Gainesville, FL, 32603, USA

Didem Pehlivanoglu, Tian Lin, Farha Deceus, Amber Heemskerk, Natalie C. Ebner & Brian S. Cahill

Department of Aging and Geriatric Research, Institute on Aging, University of Florida, Gainesville, USA

Natalie C. Ebner

Florida Institute for Cybersecurity, University of Florida, Gainesville, USA

Evelyn F. McKnight Brain Institute, University of Florida, Gainesville, USA

You can also search for this author in PubMed   Google Scholar

Contributions

BC, NCE, and FD designed the study, developed the stimuli, and collected the data. DP, TL, NCE, and BC developed the formal analytical strategy. DP processed the data, conducted analyses, and reported the findings. DP, BC, FD, TL, and NCE wrote the Methods. DP, BC, and NCE wrote the introduction and the discussion. All authors contributed to manuscript conceptualization and editing and approved the final manuscript.

Corresponding author

Correspondence to Didem Pehlivanoglu .

Ethics declarations

Ethics approval and consent to participate.

The Project was approved by University of Florida Institutional Review Board (IRB) (approval #201902194, #20200960). Electronic informed consent was obtained at enrollment from all participants using an IRB-approved consent form.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file1. appendix a:.

A full set of the news articles used in the current project.

Additional file 2. Appendix B:

Confidence ratings and sharing responses for the news articles.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Pehlivanoglu, D., Lin, T., Deceus, F. et al. The role of analytical reasoning and source credibility on the evaluation of real and fake full-length news articles. Cogn. Research 6 , 24 (2021). https://doi.org/10.1186/s41235-021-00292-3

Download citation

Received : 01 August 2020

Accepted : 17 March 2021

Published : 31 March 2021

DOI : https://doi.org/10.1186/s41235-021-00292-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Analytical reasoning
• Source credibility
• Dual-process theory
• Naturalistic decision-making

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 27 March 2024

Tweeting your research paper boosts engagement but not citations

• Bianca Nogrady

You can also search for this author in PubMed   Google Scholar

Even before complaints about X’s declining quality, posting a paper on the social-media platform did not lead to a boost in citations. Credit: Matt Cardy/Getty

Posting about a research paper on social-media platform X (formerly known as Twitter) doesn’t translate into a bump in citations, according to a study that looked at 550 papers.

The finding comes as scientists are moving away from the platform in the wake of changes after its 2022 purchase by entrepreneur Elon Musk.

An international group of 11 researchers, who by the end of the experiment had between them nearly 230,000 followers on X, examined whether there was evidence that posting about a paper would increase its citation rate.

“There certainly is a correlation, and that’s been found in a lot of papers. But very few people have ever looked to see whether there’s any experimental causation,” says Trevor Branch, a marine ecologist at the University of Washington in Seattle and lead author on the paper, published in PLoS ONE last week 1 .

Every month for ten months, each researcher was allocated a randomly selected primary research article or review from a journal of their choice to post about on their personal account. Four randomly chosen articles from the same edition of the journal served as controls, which the researchers did not post about. They conducted the experiment in the period before Elon Musk took ownership of what was then known as Twitter and complaints of its declining quality increased.

‘Nail in the coffin’

Three years after the initial posts, the team compared the citation rates for the 110 posted articles with those of the 440 control articles, and found no significant difference. The researchers did acknowledge that their followers might not have been numerous enough to detect a statistically significant effect on citations.

The rate of daily downloads for the posted papers was nearly fourfold higher on the day that they were shared, compared with controls. Shared papers also had significantly higher accumulated Altmetric scores both 30 days and three years after the initial post. Calculated by London-based technology company Digital Science, an Altmetric score, says Branch, is a measure of how many people have looked at a paper and are talking about it, but it’s not a reliable indicator of a paper’s scientific worth. “It’s thoroughly biased by how many people with large followings tweet about it,” he says.

The findings echo those of information scientist Stefanie Haustein at the University of Ottawa, whose 2013 study 2 found a low correlation between posts and citations.

Haustein says the problem with using posts as a metric is that, even a decade ago, there was a lot of noise in the signal.

“We actually showed that a lot of the counts on Twitter you would get were bots, it wasn’t even humans,” says Haustein, who wasn’t involved in the new study.

She says the more recent departure of scientists from the platform has been the final nail in the coffin of the idea that posting could increase citations.

doi: https://doi.org/10.1038/d41586-024-00922-y

Branch, T. A. et al. PLoS ONE 19 , e0292201 (2024).

Article   PubMed   Google Scholar  

Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M. & Larivière, V. J. Assoc. Inf. Sci. Technol. 65, 656–669 (2014).

Article   Google Scholar  

Download references

Reprints and permissions

Related Articles

• Communication
• Scientific community

Divas, captains, ghosts, ants and bumble-bees: collaborator attitudes explained

Career Column 15 MAR 24

Three actions PhD-holders should take to land their next job

Career Column 13 MAR 24

This geologist communicates science from the ski slopes

Career Q&A 11 MAR 24

The corpse of an exploded star and more — March’s best science images

News 28 MAR 24

How OpenAI’s text-to-video tool Sora could change science – and society

News 12 MAR 24

Giant plume of plasma on the Sun’s surface and more — February’s best science images

News 01 MAR 24

Will the Gates Foundation’s preprint-centric policy help open access?

News 04 APR 24

The EU’s ominous emphasis on ‘open strategic autonomy’ in research

Editorial 03 APR 24

Time to sound the alarm about the hidden epidemic of kidney disease

Global Scientist Interdisciplinary Forum & Recruitment

Southern University of Science and Technology, School of Medicine

Shenzhen, Guangdong, China

Research Associate - Neuroscience and Respiratory Physiology

Houston, Texas (US)

Baylor College of Medicine (BCM)

Histology Laboratory Manager

Postdoctoral scholar - research-pediatrics.

Memphis, Tennessee

The University of Tennessee Health Science Center (UTHSC)

Postdoctoral Scholar

Postdoctoral Scholar - PHAST Investigate Novel Roles of Linear Deubiquitinase Otulin In NeuroDegenerato

Memphis, Tennessee (US)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Help | Advanced Search

Computer Science > Computation and Language

Title: realm: reference resolution as language modeling.

Abstract: Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user's screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underutilized. This paper demonstrates how LLMs can be used to create an extremely effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

The Creativity Decline: Evidence from US Patents

Working Paper 2024-008A by Aakash Kalyani

Economists have long struggled to understand why aggregate productivity growth has dropped in recent decades while the number of new patents filed has steadily increased. I offer an explanation for this puzzling divergence: the creativity embodied in US patents has dropped dramatically over time. To separate creative from derivative patents, I develop a novel, text-based measure of patent creativity: the share of technical terminology that did not appear in previous patents. I show that only creative and not derivative patents are associated with significant improvements in firm level productivity. Using the measure, I show that inventors on average file creative patents upon entry, and file derivative patents with more experience. I embed this life-cycle of creativity in a growth model with endogenous creation and imitation of technologies. In this model, falling population growth explains 27% of the observed decline in patent creativity, 30% of the slowdown in productivity growth, and 64% of the increase in patenting.

Read Full Text

https://doi.org/10.20955/wp.2024.008

• Subscribe to Our Newsletter

Related Links

• Economist Pages
• JEL Classification System
• Fed In Print

SUBSCRIBE TO THE RESEARCH DIVISION NEWSLETTER

Research division.

• Legal and Privacy

One Federal Reserve Bank Plaza St. Louis, MO 63102

Information for Visitors

Accessibility Links

• Skip to content
• Skip to search IOPscience
• Skip to Journals list
• Accessibility help
• Accessibility Help

Click here to close this panel.

The American Astronomical Society (AAS) , established in 1899 and based in Washington, DC, is the major organization of professional astronomers in North America. Its membership of about 7,000 individuals also includes physicists, mathematicians, geologists, engineers, and others whose research and educational interests lie within the broad spectrum of subjects comprising contemporary astronomy. The mission of the AAS is to enhance and share humanity's scientific understanding of the universe.

The Institute of Physics (IOP) is a leading scientific society promoting physics and bringing physicists together for the benefit of all. It has a worldwide membership of around 50 000 comprising physicists from all sectors, as well as those with an interest in physics. It works to advance physics research, application and education; and engages with policy makers and the public to develop awareness and understanding of physics. Its publishing company, IOP Publishing, is a world leader in professional scientific communications.

Anthropogenic Coal Ash as a Contaminant in a Micro-meteoritic Underwater Search

Patricio A. Gallardo 1

Published October 2023 • © 2023. The Author(s). Published by the American Astronomical Society. Research Notes of the AAS , Volume 7 , Number 10 Citation Patricio A. Gallardo 2023 Res. Notes AAS 7 220 DOI 10.3847/2515-5172/ad03f9

Article metrics

11951 Total downloads

Share this article

Author affiliations.

1 KICP, University of Chicago, Chicago, USA

Patricio A. Gallardo https://orcid.org/0000-0001-9731-3617

• Received October 2023
• Accepted October 2023
• Published October 2023

Meteorite composition ; Micrometeorites ; Interdisciplinary astronomy

Receive alerts on all new research papers in American Astronomical Society ( A A S  ) journals as soon as they are published. Select your desired journals and corridors below. You will need to select a minimum of one corridor.

Please note, The Planetary Science Journal (PSJ) does not currently use the corridors.

What are corridors?

Chemical composition for spherules recovered from the search area of CNEOS 2014-01-08 in the Pacific Ocean has been recently released. A three-order of magnitude difference from CI-chondrites has been identified for elements beryllium, lanthanum and uranium in five samples. The lack of consensus regarding atmospheric survival and precision of path estimates motivate an examination of possible contaminants. Contents of nickel, beryllium, lanthanum and uranium are examined in the context of a known anthropogenic source of contamination, and found to be consistent with coal ash as suggested from a publicly available coal chemical composition database (COALQUAL). The meteoritic origin is disfavored.

Export citation and abstract BibTeX RIS

• NASA ADS Record
• About Related Links

Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

1. Introduction

An expedition to locate micrometeoritic fragments in the search area of CNEOS 2014-01-08 has been announced and conducted in the South Pacific Ocean (Loeb 2022 and Loeb et al. 2023 , L23 hereafter). A magnetic sled was used to retrieve magnetic material. Round objects in the ranges from hundreds of microns to millimeter scales have been reported. It has been suggested that the high concentration of beryllium, lanthanum and uranium in five samples (named BeLaU for short) could be anomalous as compared to CI-chondrite abundances ( L23 ). However, few comparisons to contaminants have been conducted to discard the null hypothesis of terrestrial contamination. The lack of consensus regarding the atmospheric survival and precision in the determination of the path of CNEOS 2014-01-08 (Vaubaillon 2022 ; Brown & Borovička 2023 ) motivate a discussion of possible sources of contamination from terrestrial sources.

Multiple reports during the past century have discussed anthropogenic contaminants in samples containing magnetic spherules from microns to millimeter scales in size (Handy & Davidson 1953 ; Oldfield et al. 1978 ; Goldberg et al. 1981 ; Deuser et al. 1983 ; Locke & Bertine 1986 ; Wik & Renberg 1991 ). Most notably, in 1976, another naval expedition in the Gulf of Mexico found large numbers of magnetic spherules from anthropogenic sources in seawater (Doyle et al. 1976 ). Chemical composition analyses revealed consistency with coal fly ash, a waste product of the combustion of coal in power plants and steam engines.

In this document, the chemical composition of the five spherules labeled BeLaU in L23 is analyzed in light of a known source of contamination from anthropogenic origin such as coal fly ash. Iron content is compared to previously published data from a naval expedition, which collected magnetic spherules as presented in Doyle et al. ( 1976 ) and to the iron contents in fly ash retrieved from a real power plant as in Hock & Lichtman ( 1983 ). The contents of beryllium, lanthanum, uranium and nickel are compared to expected abundances using publicly available measurements from coal quality data maintained by the USGS as in Palmer et al. ( 2015 ). This document is organized as follows: Section 1 gives a brief summary of the expedition, findings, and a description of this work. Section 2 discusses the methods used to compare abundances. Section 3 describes the results. Section 4 concludes.

2.1. Iron Composition

The iron content of coal ash has been documented in the context of coal quality control and commercial iron sourcing. Although typical values for the iron content in coal ash range 20%, with a variance of several tens of percent (Myers et al. 1973 ), higher values can be obtained if the ash is magnetically selected (Murtha & Burnet 1978 ). Spherule size is another source of bias as discussed in Czech ( 2022 ). Iron content in a real power plant was presented in Hock & Lichtman ( 1983 ), where samples were collected at the plant smokestack, reporting 35% iron concentration with a standard deviation of 30%.

In one report from a naval expedition (Doyle et al. 1976 ), seawater was microfiltered and the resulting spherules were magnetically identified. In this experiment, spherules were identified as athropogenic fly ash via their composition. Due to the similarity of the experiment presented in Doyle et al. ( 1976 ) and in Loeb ( 2022 ), the iron content reported in Doyle et al. ( 1976 ) is used to compare the range of iron abundances, considering that a variation of several tens of percent is expected from practical ash.

2.2. Beryllium, Lanthanum, Uranium and Nickel Content

Publicly available coal quality data COALQUAL summarized in Palmer et al. ( 2015 ) is used to obtain the range for the concentrations of five elements: nickel, beryllium, lanthanum and uranium in coal ash. Coal quality databases report trace abundances according to the formula

The abundance reported in Table 1 in Doyle et al. ( 1976 ) has a mean value of 68%, while the iron abundance in the BeLaU sample has a mean of 51%. In consistency with Hock & Lichtman ( 1983 ).

3.2. Nickel

Nickel content has been pointed out as a discriminator between coal ash and meteoritic material (Handy & Davidson 1953 ). Table 1 in Doyle et al. ( 1976 ) found nickel concentrations of order 0.04% in fly ash. The nickel concentrations in L23 are of order 200 ppm (0.02%) or lower. Which puts the nickel content in the same order of magnitude of Doyle et al. ( 1976 ).

In addition, COALQUAL data as described in Section 2 is used as a comparison. Figure 1 (bottom right) shows in green the concentrations of the five BeLaU samples, the histogram shows the expected nickel concentration from the COALQUAL database. Nickel content is in consistency with ash from coal.

Figure 1.  Beryllium, lanthanum, uranium and nickel concentrations in BeLaU (green) samples. Histogram shows frequencies obtained from COALQUAL. Concentrations are within expectation for all elements.

Download figure:

3.3. Beryllium, Lanthanum and Uranium

Figure 1 shows in green the concentrations for the five BeLaU samples, with the expected histogram (in black) of the concentrations from coal ash for beryllium, lanthanum and uranium. COALQUAL data shows that all samples are in the expected range, in consistency with coal ash, and with: Headlee & Hunter ( 1953 ), and Zielinski & Finkelman ( 1997 ).

4. Conclusion

A compositional comparison of five samples collected from the Pacific Ocean has been presented. The content of iron and nickel have been compared to a previous report of an ocean expedition, which collected water samples using microfilters and collected spherical magnetic objects. The contents of beryllium, lanthanum, and uranium were compared to a publicly available database of coal composition.

Iron content is found to be consistent from previous reports of coal ash contamination. Nickel, beryllium, lanthanum, and uranium concentrations are found to be consistent with expectations from coal ash from a coal chemical composition database. Fly ash resolves the three-order of magnitude difference from comparisons to CI-chondrites. The meteoritic origin is disfavored.

• a. Send us an email
• b. Anonymous form
• Buyer's Guide
• Upcoming Products
• Tips / Contact Us
• Podcast Instagram Facebook Twitter Mastodon YouTube Notifications RSS Newsletter

Apple Researchers Reveal New AI System That Can Beat GPT-4

Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.

Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple's ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.

ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.

ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.

Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.

Get weekly top MacRumors stories in your inbox.

Top Rated Comments

enabling users to use pronouns and other indirect references in conversation without confusion.

It's good if AI understands "Can you repeat that?" properly. /thread

[HEADING=2]Apple Researchers Reveal[/HEADING]

Popular Stories

iPhone 16 Pro Expected Later This Year With These 12 New Features

Alleged iOS 18 Design Resource Reveals visionOS-Like Redesign [Updated]

Apple Exploring 'Mobile Robot' That 'Follows Users Around Their Homes'

Apple Card Savings Account to Receive First-Ever Interest Rate Decrease

What to Expect From iOS 17.5

Apple Says iPhone 6 Plus Now 'Obsolete' and iPad Mini 4 Now 'Vintage'

Best Buy Introduces All-Time Low Prices on Apple's M3 MacBook Pro for Members

Next Article

Our comprehensive guide highlighting every major new addition in iOS 17, plus how-tos that walk you through using the new features.

App Store changes for the EU, new emoji, Podcasts transcripts, and more.

Get the most out your iPhone 15 with our complete guide to all the new features.

A deep dive into new features in macOS Sonoma, big and small.

Revamped models with OLED displays, M3 chip, and redesigned Magic Keyboard accessory.

Updated 10.9-inch model and new 12.9-inch model, M2 chip expected.

Apple's annual Worldwide Developers Conference will kick off with a keynote on June 10.

Expected to see new AI-focused features and more. Preview coming at WWDC in June with public release in September.

Other Stories

2 days ago by MacRumors Staff

6 days ago by Tim Hardwick

6 days ago by Juli Clover

You are using an outdated browser. Please upgrade your browser to improve your experience.

Apple AI research: ReALM is smaller, faster than GPT-4 when parsing contextual data

Apple is working to bring AI to Siri

Artificial Intelligence research at Apple keeps being published as the company approaches a public launch of its AI initiatives in June during WWDC . There has been a variety of research published so far, including an image animation tool .

The latest paper was first shared by VentureBeat . The paper details something called ReALM — Reference Resolution As Language Modeling.

Having a computer program perform a task based on vague language inputs, like how a user might say "this" or "that," is called reference resolution. It's a complex issue to solve since computers can't interpret images the way humans can, but Apple may have found a streamlined resolution using LLMs.

When speaking to smart assistants like Siri , users might reference any number of contextual information to interact with, such as background tasks, on-display data, and other non-conversational entities. Traditional parsing methods rely on incredibly large models and reference materials like images, but Apple has streamlined the approach by converting everything to text.

Apple found that its smallest ReALM models performed similarly to GPT-4 with much fewer parameters, thus better suited for on-device use. Increasing the parameters used in ReALM made it substantially outperform GPT-4.

One reason for this performance boost is GPT-4's reliance on image parsing to understand on-screen information. Much of the image training data is built on natural imagery, not artificial code-based web pages filled with text, so direct OCR is less efficient.

Converting an image into text allows ReALM to skip needing these advanced image recognition parameters, thus making it smaller and more efficient. Apple also avoids issues with hallucination by including the ability to constrain decoding or use simple post-processing.

For example, if you're scrolling a website and decide you'd like to call the business, simply saying "call the business" requires Siri to parse what you mean given the context. It would be able to "see" that there's a phone number on the page that is labeled as the business number and call it without further user prompt.

Apple is working to release a comprehensive AI strategy during WWDC 2024. Some rumors suggest the company will rely on smaller on-device models that preserve privacy and security, while licensing other company's LLMs for the more controversial off-device processing filled with ethical conundrums.

Top Stories

This best-selling M3 MacBook Pro 14-inch with 16GB RAM is on sale for $1,599

What to expect from Apple's Q2 2024 earnings on May 2

iPhone 16 dummy units show off Capture button, new camera bump

Thinnest iPhone 16 display bezels still a problem for OLED suppliers

Two more new iPads spotted in regulatory database

Beyond TSMC, Apple's supply chain will be disrupted by the Taiwan earthquake

Featured deals.

Save $400 on Apple's 15-inch MacBook Air with 24GB RAM, 2TB SSD

Latest comparisons.

M3 15-inch MacBook Air vs M3 14-inch MacBook Pro — Ultimate buyer's guide

M3 MacBook Air vs M1 MacBook Air — Compared

M3 MacBook Air vs M2 MacBook Air — Compared

Latest news.

Apple Robots, Apple anniversaries, plus Spatial Video pros & cons, on the AppleInsider Podcast

On this week's episode of the AppleInsider Podcast, your hosts discuss the new Apple Robot rumors, reminisce about Apple's founding and the 14th anniversary of the iPad, and so much more!

Samsung overtakes Apple to become world's leading smartphone vendor

It's less a case of popularity, and more about release cycles, but Samsung is again the number one smartphone manufacturer in the world.

Another two unannounced iPads have now been listed on an Indian regulator's database, meaning the expected iPad Pro and iPad Air models are likely to be released soon.

How to fix corrupted DaVinci Resolve projects

Most video creators are familiar with the dreaded "Media Offline" warning and the panic it triggers. For users of DaVinci Resolve, the fix usually takes just a few clicks, but sometimes relinking clips seems impossible, and a project appears to be gone forever.

Apple lays off 600 employees, mostly from Apple Car project

After Apple killed its ambitious Apple Car project and microLED Apple Watch display dreams, the company has laid off over 600 employees.

Google's Apple-friendly Find My Devices network launching in April

Apple and Google have worked together to get an interoperability standard off the ground for tracking devices, and Google's Find My Devices network is ready to launch.

Apple offers a sneak peek into new developer channel with YouTube trailer

Apple's new developer channel on YouTube has released a short video promoting developing for Apple platforms ahead of its Worldwide Developer Conference.

Sponsored Content

Bluetti AC240 portable power station pushes the boundaries with IP65 waterproof rating

Rugged outdoor adventures require tough equipment capable of shrugging off the elements and providing reliability when the weather turns. Bluetti's done just that with the new AC240 weatherproof portable power station, available now with an early bird discount.

Russian antitrust regulator asks Apple about banking apps while ignoring Ukraine war

Russia's Federal Antimonopoly Service has asked Apple why Russian users cannot access full banking and payment services, while seemingly ignoring how banks in the country were sanctioned over the Ukraine war.

Google could charge Apple users for AI tools in iOS 18

Rumors suggest Google is looking to offer premium generative AI features just as Apple is allegedly planning an AI App Store for iOS 18.

Latest Videos

How to turn off Apple's Journal 'Discoverable by Others' setting that's enabled by default

The best Thunderbolt 4 docks and hubs you can buy for your Mac

Apple Ring rumors & research - what you need to know about Apple's next wearable

Latest reviews.

TP-Link Tapo Indoor cameras review: affordable HomeKit options with in-app AI tools

ShiftCam LensUltra Deluxe Kit review: Upgrade your iPhone photo shooting game

Keychron Q1 Max review: cushy, comfortable, costly

{{ title }}

{{ summary }}