how to write a systematic literature review psychology

• Academic Resources
• Grey Literature: Resources
• Conducting a Systematic Review
• "How to" videos: searching
• Reference Management Software: Zotero This link opens in a new window
• Further Support

Systematic Reviews: Introduction

• What is a systematic review?
• How does it differ from a systematic literature review?

A systematic review is a review of research literature (often limited to randomised controlled trials) that is searched for in a systematic or exact manner, and only includes studies that have met strict inclusion criteria. The methodology of the review often follows a internationally recognized systematic review standard or guideline. 

PRISMA definition

"A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review." (PRISMA, 2015)

A systematic literature search is a literature review which demonstrates that you have compiled a list of appropriate search terms and includes the structure of your search history, which provides the evidence on which your assignment is based.  You may be told that you need to conduct a systematic review when in fact you just need to perform a literature search in a systematic manner. 

This is a less rigorous process than a systematic review. A systematic review usually covers a wider scope; you would be expected to look at all the available research in the area in question, systematically searching multiple academic databases.

If you are unsure about the differences between a systematic review and a literature review take a look at this guide:  What’s in a Name? The difference between a Systematic Review and a Literature Review and Why it Matters .

Systematic reviews: your research question

Your research question

• Inclusion and exclusion criteria
• PICO framework
• PICO: example

Where possible, select a research topic that you have some basic knowledge in, or are familiar with in.  If this is not possible, seek guidance from your supervisor on the most relevant terminology for your chosen topic.  This will make it easier to devise your research question, ensuring you are using the most appropriate and relevant phrasing.This is especially important when you come to build your search strategy, as the wording of your research question research question will frame your initial search. 

The search terms, phrases, words and subject headings you use in your systematic search play a vital role in your systematic review, because they determine the number and range of results you will get from each database. I f your question is too wide-reaching so will your search, and you will be overwhelmed with  an unmanageable number of search results. You also have to think very carefully about the limitations of your search - such as whether or not you will have a specific date range or limit to a particular study design.  Remember, the clue is in the name - be  systematic  in your approach. 

Inclusion and exclusion criteria 

Before you start to build your search strategy you will need to decide on your review's inclusion and exclusion criteria. This will have a bearing on your search strategy, and potentially determine which studies your systematic search captures. Systematic reviews are measured against a set of specific criteria outlined at the start.  Your inclusion and exclusion criteria dictates which studies will be included in your systematic review (they meet all aspects of your inclusion criteria) and those that will be excluded (does not match your inclusion criteria fully, or meets the exclusion criteria).  You need to think very carefully about the limitations of your research - will you limit to a specific date range or to a particular study design for example.  Remember, the clue is in the name - be  systematic  in your approach. 

Research question:

"For UK children under 5 years of age with a suspected diagnosis of ADHD, what are the recommended clinical assessments used for diagnosis?" 

Inclusion criteria:

• children 0-5 years old 
• suspected ADHD
• limited to clinical assessments specifically, for diagnosis
• UK population studies only

Exclusion criteria:

• any member of the population over the age of 5 
• animal studies
• pre-determined diagnosis of ADHD 
• assessments not defined as "clinical" 
• studies of populations outside of the UK

PICO Framework

In order to structure your search strategy  it can be helpful to use a structured framework.They are designed to help you formulate your research question, search strategy, and inclusion and exclusion criteria. The PICO framework is a popular framework for physiological and health sciences systematic reviews.

PICO stands for:

Population / Patient

Intervention 

You do not have to structure your research to fit into all of the above - it is not prescriptive, but merely a framework to guide you in building your search strategy. Your research question may not have a defined comparative intervention, for example, so you may only use PIO for your framework. 

PICO example

Research question: What impact does Cognitive Behavioural Therapy (CBT) versus pharmaceutical therapy have as treatments for diagnosed anxiety in children  in reducing their anxiety?

Systematic Reviews: the search

• Your search strategy
• Where to search
• Search techniques
• PRESS checklist
• Sample strategy

Your search strategy - where to start

The search terms, phrases, words and subject headings you use in your search play a vital role in your systematic review, because they determine the number and range of results you will get from each database. 

Start with your research question to pick out the  keywords  you will use as the basis of your search. A common practice in systematic review searches is to refer back to the framework you used to structure your research question, such as the PICO framework . This framework gives a clear picture of not only the terms you want to use in your search, but also how you will combine or connect them together in the database.

Using PICO framework for your search 

Research question: Is Cognitive Behavioural Therapy (CBT) more effective than pharmaceutical therapy as a treatment for diagnosed anxiety in teenagers aged 13-18 in managing symptoms of anxiety?

Framework  

Search terms identified: 

• teen/teenager
• 13-18 year olds
• anxiety 
• CBT/cognitive behavioural the rapy
• pharmaceutical/medication
• Improved   management of symptoms

Where to search - Databases

We have grouped the key databases and online resources for psychology in the Key Resources tab . Your dissertation supervisor may also suggest or advise you on the various databases suitable to your search. Some of the most commonly searched databases include: PsycINFO, Web of Science, Cochrane, Scopus, and PubMed. The links for these can be found in the Key Resources tab. 

Where to search - Grey Literature

Depending on your research, you may also need to search grey literature sources in addition to academic database collections. Have a look at the grey literature tab  on the left hand side more information about what defines grey literature and the sources you can search. 

Search techniques 

Consider your keywords carefully 

It's often helpful to look at the abstracts and reference lists of studies or papers you have already engaged with on your topic for a sense of what terminology and specific words/phrases are most common. 

Once you have made a comprehensive list of all the keywords you want to include in your search you need to combine them using what are known as Boolean operators or search connectors. 

Boolean search operators 

In order to combine multiple facets of your search together in a database you will need to use the search operators: AND, OR, NOT. Using operators is common in systematic reviews, and improves the efficiency of your search resulting in more relevant results. Watch the Boolean searching videos to learn more about how to use operators.

Proximity operators 

Using proximity operators in your search allows you to  search for two or more words that occur within a certain number of words from each other. Proximity operators can be helpful when you don't want to limit yourself to searching an exact phrase, for example "diagnosed anxiety", but where you still want to refine your search to a degree. Each database will have its own method of proximity searching, but generally your search looks something like: diagnosed NEAR/3 anxiety - NEAR signifying near to, and 3 signifying the number of words that can appear between "diagnosed" and "anxiety". To ensure proximity searching is available in the database you are searching, have a look for their "search tips" or "search help" page. 

PRESS (Peer Review of Electronic Search Strategies)

The PRESS checklist is a comprehensive evidence-based checklist that does just that. It covers six key aspects of search strategies: 

• the translation of the research question into a search strategy
• the use of any Boolean or proximity operators
• the keywords used 
• spelling, syntax, and commands
• database filters / limits intended for use
• database Subject headings intended for use not essential, and may not apply to your strategy)

Each aspect above has a set of questions to check against your search strategy. The full checklist can be downloaded here . 

Research question

Is Cognitive Behavioural Therapy (CBT) more effective than pharmaceutical therapy as a treatment for diagnosed anxiety in teenagers aged 13-18 in managing symptoms of anxiety?

Step 1: Framework

Step 2: Identify search terms

Step 3: Build search strategy with Boolean operators / commands / proximity operators

Systematic Reviews: guidelines and protocols

• Standards and guidelines
• Cochrane handbook
• Choosing a standard/guideline

Standards and guidelines 

These can serve as guidelines for the structure of your review, from what headings/sections to include, (such as methods, search strategy, etc) to best practice protocols for search strategies, reporting, etc. 

​ The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review. 

PRISMA standards

PRISM A  (Preferred Reporting Items for Systematic Reviews and  Meta-Analyses )  is an evidence-based protocol for reporting on systematic reviews and meta-analyses. The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review. The PRISMA checklist makes this easy, providing a checklist of criteria to be adhered to for each of the sections of your review, for example: introduction, methods, results, etc.

Useful PRISMA links: 

• PRISMA Checklist  (checklist of sections to include in your systematic review, eg: methods, results, etc.) 
• PRISMA flow chart   (charts the number of records identified, included and excluded, and the reasons for exclusions.)
• Citing PRISMA

Cochrane Handbook 

Cochrane Handbook for Systematic Reviews of Interventions  

Choosing a systematic review standard/guideline

The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review, but it is also helpful to read through the Cochrane Handbook of Systematic Review, particularly if your research topic spans both medical and human sciences.

• << Previous: Grey Literature: Resources
• Next: "How to" videos: searching >>
• Last Updated: Nov 24, 2023 10:54 AM
• URL: https://guides.lib.sussex.ac.uk/psychology

• Search This Site All UCSD Sites Faculty/Staff Search Term
• Contact & Directions
• Climate Statement
• Cognitive Behavioral Neuroscience
• Cognitive Psychology
• Developmental Psychology
• Social Psychology
• Adjunct Faculty
• Non-Senate Instructors
• Researchers
• Psychology Grads
• Affiliated Grads
• New and Prospective Students
• Honors Program
• Experiential Learning
• Programs & Events
• Psi Chi / Psychology Club
• Prospective PhD Students
• Current PhD Students
• Area Brown Bags
• Colloquium Series
• Anderson Distinguished Lecture Series
• Speaker Videos
• Undergraduate Program
• Academic and Writing Resources

Writing Research Papers

• Writing a Literature Review

When writing a research paper on a specific topic, you will often need to include an overview of any prior research that has been conducted on that topic.  For example, if your research paper is describing an experiment on fear conditioning, then you will probably need to provide an overview of prior research on fear conditioning.  That overview is typically known as a literature review.  

Please note that a full-length literature review article may be suitable for fulfilling the requirements for the Psychology B.S. Degree Research Paper .  For further details, please check with your faculty advisor.

Different Types of Literature Reviews

Literature reviews come in many forms.  They can be part of a research paper, for example as part of the Introduction section.  They can be one chapter of a doctoral dissertation.  Literature reviews can also “stand alone” as separate articles by themselves.  For instance, some journals such as Annual Review of Psychology , Psychological Bulletin , and others typically publish full-length review articles.  Similarly, in courses at UCSD, you may be asked to write a research paper that is itself a literature review (such as, with an instructor’s permission, in fulfillment of the B.S. Degree Research Paper requirement). Alternatively, you may be expected to include a literature review as part of a larger research paper (such as part of an Honors Thesis). 

Literature reviews can be written using a variety of different styles.  These may differ in the way prior research is reviewed as well as the way in which the literature review is organized.  Examples of stylistic variations in literature reviews include: 

• Summarization of prior work vs. critical evaluation. In some cases, prior research is simply described and summarized; in other cases, the writer compares, contrasts, and may even critique prior research (for example, discusses their strengths and weaknesses).
• Chronological vs. categorical and other types of organization. In some cases, the literature review begins with the oldest research and advances until it concludes with the latest research.  In other cases, research is discussed by category (such as in groupings of closely related studies) without regard for chronological order.  In yet other cases, research is discussed in terms of opposing views (such as when different research studies or researchers disagree with one another).

Overall, all literature reviews, whether they are written as a part of a larger work or as separate articles unto themselves, have a common feature: they do not present new research; rather, they provide an overview of prior research on a specific topic . 

How to Write a Literature Review

When writing a literature review, it can be helpful to rely on the following steps.  Please note that these procedures are not necessarily only for writing a literature review that becomes part of a larger article; they can also be used for writing a full-length article that is itself a literature review (although such reviews are typically more detailed and exhaustive; for more information please refer to the Further Resources section of this page).

Steps for Writing a Literature Review

1. Identify and define the topic that you will be reviewing.

The topic, which is commonly a research question (or problem) of some kind, needs to be identified and defined as clearly as possible.  You need to have an idea of what you will be reviewing in order to effectively search for references and to write a coherent summary of the research on it.  At this stage it can be helpful to write down a description of the research question, area, or topic that you will be reviewing, as well as to identify any keywords that you will be using to search for relevant research.

2. Conduct a literature search.

Use a range of keywords to search databases such as PsycINFO and any others that may contain relevant articles.  You should focus on peer-reviewed, scholarly articles.  Published books may also be helpful, but keep in mind that peer-reviewed articles are widely considered to be the “gold standard” of scientific research.  Read through titles and abstracts, select and obtain articles (that is, download, copy, or print them out), and save your searches as needed.  For more information about this step, please see the Using Databases and Finding Scholarly References section of this website.

3. Read through the research that you have found and take notes.

Absorb as much information as you can.  Read through the articles and books that you have found, and as you do, take notes.  The notes should include anything that will be helpful in advancing your own thinking about the topic and in helping you write the literature review (such as key points, ideas, or even page numbers that index key information).  Some references may turn out to be more helpful than others; you may notice patterns or striking contrasts between different sources ; and some sources may refer to yet other sources of potential interest.  This is often the most time-consuming part of the review process.  However, it is also where you get to learn about the topic in great detail.  For more details about taking notes, please see the “Reading Sources and Taking Notes” section of the Finding Scholarly References page of this website.

4. Organize your notes and thoughts; create an outline.

At this stage, you are close to writing the review itself.  However, it is often helpful to first reflect on all the reading that you have done.  What patterns stand out?  Do the different sources converge on a consensus?  Or not?  What unresolved questions still remain?  You should look over your notes (it may also be helpful to reorganize them), and as you do, to think about how you will present this research in your literature review.  Are you going to summarize or critically evaluate?  Are you going to use a chronological or other type of organizational structure?  It can also be helpful to create an outline of how your literature review will be structured.

5. Write the literature review itself and edit and revise as needed.

The final stage involves writing.  When writing, keep in mind that literature reviews are generally characterized by a summary style in which prior research is described sufficiently to explain critical findings but does not include a high level of detail (if readers want to learn about all the specific details of a study, then they can look up the references that you cite and read the original articles themselves).  However, the degree of emphasis that is given to individual studies may vary (more or less detail may be warranted depending on how critical or unique a given study was).   After you have written a first draft, you should read it carefully and then edit and revise as needed.  You may need to repeat this process more than once.  It may be helpful to have another person read through your draft(s) and provide feedback.

6. Incorporate the literature review into your research paper draft.

After the literature review is complete, you should incorporate it into your research paper (if you are writing the review as one component of a larger paper).  Depending on the stage at which your paper is at, this may involve merging your literature review into a partially complete Introduction section, writing the rest of the paper around the literature review, or other processes.

Further Tips for Writing a Literature Review

Full-length literature reviews

• Many full-length literature review articles use a three-part structure: Introduction (where the topic is identified and any trends or major problems in the literature are introduced), Body (where the studies that comprise the literature on that topic are discussed), and Discussion or Conclusion (where major patterns and points are discussed and the general state of what is known about the topic is summarized)

Literature reviews as part of a larger paper

• An “express method” of writing a literature review for a research paper is as follows: first, write a one paragraph description of each article that you read. Second, choose how you will order all the paragraphs and combine them in one document.  Third, add transitions between the paragraphs, as well as an introductory and concluding paragraph. 1
• A literature review that is part of a larger research paper typically does not have to be exhaustive. Rather, it should contain most or all of the significant studies about a research topic but not tangential or loosely related ones. 2   Generally, literature reviews should be sufficient for the reader to understand the major issues and key findings about a research topic.  You may however need to confer with your instructor or editor to determine how comprehensive you need to be.

Benefits of Literature Reviews

By summarizing prior research on a topic, literature reviews have multiple benefits.  These include:

• Literature reviews help readers understand what is known about a topic without having to find and read through multiple sources.
• Literature reviews help “set the stage” for later reading about new research on a given topic (such as if they are placed in the Introduction of a larger research paper). In other words, they provide helpful background and context.
• Literature reviews can also help the writer learn about a given topic while in the process of preparing the review itself. In the act of research and writing the literature review, the writer gains expertise on the topic .

Downloadable Resources

• How to Write APA Style Research Papers (a comprehensive guide) [ PDF ]
• Tips for Writing APA Style Research Papers (a brief summary) [ PDF ]
• Example APA Style Research Paper (for B.S. Degree – literature review) [ PDF ]

Further Resources

How-To Videos     

• Writing Research Paper Videos
• UCSD Library Psychology Research Guide: Literature Reviews

External Resources

• Developing and Writing a Literature Review from N Carolina A&T State University
• Example of a Short Literature Review from York College CUNY
• How to Write a Review of Literature from UW-Madison
• Writing a Literature Review from UC Santa Cruz  
• Pautasso, M. (2013). Ten Simple Rules for Writing a Literature Review. PLoS Computational Biology, 9 (7), e1003149. doi : 1371/journal.pcbi.1003149

1 Ashton, W. Writing a short literature review . [PDF]     

2 carver, l. (2014).  writing the research paper [workshop]. , prepared by s. c. pan for ucsd psychology.

Back to top

• Research Paper Structure
• Formatting Research Papers
• Using Databases and Finding References
• What Types of References Are Appropriate?
• Evaluating References and Taking Notes
• Citing References
• Writing Process and Revising
• Improving Scientific Writing
• Academic Integrity and Avoiding Plagiarism
• Writing Research Papers Videos

Psychology: Systematic Reviews and Literature Reviews

• Find Articles
• Find Journals
• Dissertations
• Videos & More
• Systematic Reviews and Literature Reviews

Guidelines for Systematic Reviews

• PRISMA Checklist The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) has developed a 27 checklist items pertaining to the content of a systematic review and meta-analysis, which include the title, abstract, methods, results, discussion and funding.
• NIH Guidelines This website provides guidelines developed by the Institute of Medicine (IOM). more... less... In 2011, the Institute of Medicine (IOM) defined a systematic evidence review as "a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data."

Register your Systematic Review

Prospero Registry

"PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome." (Website- About)

Conducting a Literature Review

• The Literature Review (25 minutes Video - opens in a new window)) This in-depth video lecture explains how to write a Literature Review, and examines which elements are required in one. This video was created the Student Learning Centre at Massey University, Auckland.

Systematic Reviews - What to Consider

Before you Begin: 

• Did you know, it takes an average of 18 months to complete a systematic review?
• Did you know that it's recommended to have at least 3 people on a systematic review team? 
• Did you know that a systematic review answers a very specific type of research question? Is your question a good fit for a systematic review? 

Contact your librarian, Emily Hart , to learn more about systematic reviews. 

Types of Reviews

Comparison of Review Types from Cornell University Libraries

" A systematic review is a review of the literature that is conducted in a methodical manner based on a pre-specified protocol and with the aim of synthesizing the retrieved information often by means of a meta-analysis."

" A literature review is a descriptive and/or analytic summary of the existing material relating to some topic or area of study."

(Sage Research Methods Online)

Systematic Review vs. Literature Review - What's the Difference?

• << Previous: Videos & More
• Last Updated: Jan 8, 2024 2:49 PM
• URL: https://researchguides.library.syr.edu/psychology
• Print This Page

• Psychology and Psychiatry
• PsycINFO (Ovid) Tutorial
• Point of Care Resources
• Tests and Measures

Systematic Literature Review

• Referencing and citing
• Library Drop-in Sessions

 Outlined below is a systematic approach to conducting a literature review using the PICO(T) principle.

Follow the steps to:

• formulate an answerable question/issue
• refine concepts and keywords
• develop search strategies
• critically appraise literature
• complete the Research Planner

Step 1: Question Type

Step 2: pico(t) principle.

The PICO(T) principle is a framework in which to formulate a question or issue of interest. Questions should be answerable and refined, so to identify each of the components in the PICO formula. This will allow for keywords to be easily defined and more effective search results returned.

P  Population/Problem

I   Intervention

C  Comparison/Control

O  Outcome

(T) Timeframe

An answerable question would ask: In P how does I compared with C affect the O ?

Step 3: Prepare for Searching

Identify the Search Terms

• Consider what the main concepts of the question are
• Identify keywords and synonyms

 Combine your search terms using Boolean Operators

AND    Look for articles that include all identified keywords

OR      Look for articles that include any identified keywords

NOT    Exclude articles that contain this specific keyword

Step 4: Effective Searching

MeSH - Medical Subject Headings Is a controlled vocabulary and is used for the purpose of indexing journals and books. They are pre-determined concepts/terms which describe content in discipline specific, indexing databases PubMed, Medline and Cinahl databases include MeSH Headings or Cinhal Subject Headings, which can also serve as a Thesaurus. Searching the databases using MeSH terms allows for terms to be searched as a Major Concept (the search will return only records for which the subject heading is a major point of the article) and also provides the ability to Explode (expands the subject heading to retrieve all references indexed to that term, plus any narrower terms that are connected)

Discipline specific Databases

Medline:  One of the most acknowledged biomedicine databases, Medline's coverage includes basic research in the biomedical and clinical sciences as well as major health related subjects including nursing and allied health. Its records are indexed using MeSH.  PubMed is the publicly accessible version of Medline.

Cochrane Library:   P rovides access to a collection of databases that contain high-quality, independent evidence to inform health care decision-making. These databases include: Cochrane Reviews, Database of Abstracts of Reviews of Effects - DARE (Other Reviews), Clinical Trials and NHS Economic Evaluation Database.

PubMed: PubMed Clinical Queries allows you to find critically appraised information indexed in MEDLINE. PubMed uses a broad definition of systematic reviews. It includes systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, and guidelines.

Scopus:  Offers peer-reviewed journals in the scientific, technical, medical and social science fields. An excellent source for locating Authors' works.

Step 5: Critical Appraisal

Critical appraisal is applying a systematic approach to analyse and identify unbiased/biased and valid literature. A literature review should not be a list summarising each article retrieved. Instead, it should appear as organised sections into themes or concepts, synthesised and evaluated, according to the research question.

Set your criteria to assist in critically appraising the literature e.g.,

• Does the literature directly relate to the question?
• What are the strengths and limitations of the literature and how does it contribute to the study?

Read more about Critical Appraisal under the Evidence Based Nursing tab  

Use CASP (Critical Appraisal Skills Program) to analyse and critique the literature

Step 6: Research Planner

• Research Planner Use the Planner to record your search strategies etc

Use the Research Planner to assist in developing a recordable framework for the Literature Review

Tools to Manage References

Students are required to use APA Style referencing

       Recite - APA 

EndNote is software that allows you to record, organise and use references and PDF files. It assists in producing bibliographies and will generate citations and bibliographies in specific styles (e.g. APA 6th) into word documents. Read more about EndNote

• << Previous: Evidence Based Psychology
• Next: Referencing and citing >>
• Last Updated: Mar 16, 2024 12:00 PM
• URL: https://unimelb.libguides.com/psychology

• UNC Libraries
• HSL Academic Process
• Systematic Reviews
• Step 8: Write the Review

Systematic Reviews: Step 8: Write the Review

Created by health science librarians.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol
• Step 3: Conduct Literature Searches
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies

About Step 8: Write the Review

Write your review, report your review with prisma, review sections, plain language summaries for systematic reviews, writing the review- webinars.

• Writing the Review FAQs

  Check our FAQ's

   Email us

  Chat with us (during business hours)

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

In Step 8, you will write an article or a paper about your systematic review.  It will likely have five sections: introduction, methods, results, discussion, and conclusion.  You will: 

• Review the reporting standards you will use, such as PRISMA. 
• Gather your completed data tables and PRISMA chart. 
• Write the Introduction to the topic and your study, Methods of your research, Results of your research, and Discussion of your results.
• Write an Abstract describing your study and a Conclusion summarizing your paper. 
• Cite the studies included in your systematic review and any other articles you may have used in your paper. 
• If you wish to publish your work, choose a target journal for your article.

The PRISMA Checklist will help you report the details of your systematic review. Your paper will also include a PRISMA chart that is an image of your research process. 

Click an item below to see how it applies to Step 8: Write the Review.

Reporting your review with PRISMA

To write your review, you will need the data from your PRISMA flow diagram .  Review the PRISMA checklist to see which items you should report in your methods section.

Managing your review with Covidence

When you screen in Covidence, it will record the numbers you need for your PRISMA flow diagram from duplicate removal through inclusion of studies.  You may need to add additional information, such as the number of references from each database, citations you find through grey literature or other searching methods, or the number of studies found in your previous work if you are updating a systematic review.

How a librarian can help with Step 8

A librarian can advise you on the process of organizing and writing up your systematic review, including: 

• Applying the PRISMA reporting templates and the level of detail to include for each element
• How to report a systematic review search strategy and your review methodology in the completed review
• How to use prior published reviews to guide you in organizing your manuscript 

Reporting standards & guidelines

Be sure to reference reporting standards when writing your review. This helps ensure that you communicate essential components of your methods, results, and conclusions. There are a number of tools that can be used to ensure compliance with reporting guidelines. A few review-writing resources are listed below.

• Cochrane Handbook - Chapter 15: Interpreting results and drawing conclusions
• JBI Manual for Evidence Synthesis - Chapter 12.3 The systematic review
• PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses.

Tools for writing your review

• RevMan (Cochrane Training)
• Methods Wizard (Systematic Review Accelerator) The Methods Wizard is part of the Systematic Review Accelerator created by Bond University and the Institute for Evidence-Based Healthcare.
• UNC HSL Systematic Review Manuscript Template Systematic review manuscript template(.doc) adapted from the PRISMA 2020 checklist. This document provides authors with template for writing about their systematic review. Each table contains a PRISMA checklist item that should be written about in that section, the matching PRISMA Item number, and a box where authors can indicate if an item has been completed. Once text has been added, delete any remaining instructions and the PRISMA checklist tables from the end of each section.
• The PRISMA 2020 statement: an updated guideline for reporting systematic reviews The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies.
• PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews This document is intended to enhance the use, understanding and dissemination of the PRISMA 2020 Statement. Through examples and explanations, the meaning and rationale for each checklist item are presented.

The PRISMA checklist

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) is a 27-item checklist used to improve transparency in systematic reviews. These items cover all aspects of the manuscript, including title, abstract, introduction, methods, results, discussion, and funding. The PRISMA checklist can be downloaded in PDF or Word files.

• PRISMA 2020 Checklists Download the 2020 PRISMA Checklists in Word or PDF formats or download the expanded checklist (PDF).

The PRISMA flow diagram

The PRISMA Flow Diagram visually depicts the flow of studies through each phase of the review process. The PRISMA Flow Diagram can be downloaded in Word files.

• PRISMA 2020 Flow Diagrams The flow diagram depicts the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions. Different templates are available depending on the type of review (new or updated) and sources used to identify studies.

Documenting grey literature and/or hand searches

If you have also searched additional sources, such as professional organization websites, cited or citing references, etc., document your grey literature search using the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

Complete the boxes documenting your database searches,  Identification of studies via databases and registers, according to the PRISMA flow diagram instructions.  Complete the boxes documenting your grey literature and/or hand searches on the right side of the template, Identification of studies via other methods, using the steps below.

Need help completing the PRISMA flow diagram?

There are different PRISMA flow diagram templates for new and updated reviews, as well as different templates for reviews with and without grey literature searches. Be sure you download the correct template to match your review methods, then follow the steps below for each portion of the diagram you have available.

View the step-by-step explanation of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases and registers only or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases and registers only . 

View the step-by-step explanation of the grey literature & hand searching portion of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

View the step-by-step explanation of review update portion of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases and registers only or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

For more information about updating your systematic review, see the box Updating Your Review? on the Step 3: Conduct Literature Searches page of the guide.

Sections of a Scientific Manuscript

Scientific articles often follow the IMRaD format: Introduction, Methods, Results, and Discussion.  You will also need a title and an abstract to summarize your research.

You can read more about scientific writing through the library guides below.

• Structure of Scholarly Articles & Peer Review • Explains the standard parts of a medical research article • Compares scholarly journals, professional trade journals, and magazines • Explains peer review and how to find peer reviewed articles and journals
• Writing in the Health Sciences (For Students and Instructors)
• Citing & Writing Tools & Guides Includes links to guides for popular citation managers such as EndNote, Sciwheel, Zotero; copyright basics; APA & AMA Style guides; Plagiarism & Citing Sources; Citing & Writing: How to Write Scientific Papers

Sections of a Systematic Review Manuscript

Systematic reviews follow the same structure as original research articles, but you will need to report on your search instead of on details like the participants or sampling. Sections of your manuscript are shown as bold headings in the PRISMA checklist.

Refer to the PRISMA checklist for more information.

Consider including a Plain Language Summary (PLS) when you publish your systematic review. Like an abstract, a PLS gives an overview of your study, but is specifically written and formatted to be easy for non-experts to understand. 

Tips for writing a PLS:

• Use clear headings e.g. "why did we do this study?"; "what did we do?"; "what did we find?"
• Use active voice e.g. "we searched for articles in 5 databases instead of "5 databases were searched"
• Consider need-to-know vs. nice-to-know: what is most important for readers to understand about your study? Be sure to provide the most important points without misrepresenting your study or misleading the reader. 
• Keep it short: Many journals recommend keeping your plain language summary less than 250 words. 
• Check journal guidelines: Your journal may have specific guidelines about the format of your plain language summary and when you can publish it. Look at journal guidelines before submitting your article. 

Learn more about Plain Language Summaries: 

• Rosenberg, A., Baróniková, S., & Feighery, L. (2021). Open Pharma recommendations for plain language summaries of peer-reviewed medical journal publications. Current Medical Research and Opinion, 37(11), 2015–2016.  https://doi.org/10.1080/03007995.2021.1971185
• Lobban, D., Gardner, J., & Matheis, R. (2021). Plain language summaries of publications of company-sponsored medical research: what key questions do we need to address? Current Medical Research and Opinion, 1–12. https://doi.org/10.1080/03007995.2021.1997221
• Cochrane Community. (2022, March 21). Updated template and guidance for writing Plain Language Summaries in Cochrane Reviews now available. https://community.cochrane.org/news/updated-template-and-guidance-writing-plain-language-summaries-cochrane-reviews-now-available
• You can also look at our Health Literacy LibGuide:  https://guides.lib.unc.edu/healthliteracy 

How to Approach Writing a Background Section

What Makes a Good Discussion Section

Writing Up Risk of Bias

Developing Your Implications for Research Section

• << Previous: Step 7: Extract Data from Included Studies
• Next: FAQs >>
• Last Updated: Mar 28, 2024 9:43 AM
• URL: https://guides.lib.unc.edu/systematic-reviews

Search & Find

• E-Research by Discipline
• More Search & Find

Places & Spaces

• Places to Study
• Book a Study Room
• Printers, Scanners, & Computers
• More Places & Spaces
• Borrowing & Circulation
• Request a Title for Purchase
• Schedule Instruction Session
• More Services

Support & Guides

• Course Reserves
• Research Guides
• Citing & Writing
• More Support & Guides
• Mission Statement
• Diversity Statement
• Staff Directory
• Job Opportunities
• Give to the Libraries
• News & Exhibits
• Reckoning Initiative
• More About Us

• Search This Site
• Privacy Policy
• Accessibility
• Give Us Your Feedback
• 208 Raleigh Street CB #3916
• Chapel Hill, NC 27515-8890
• 919-962-1053

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved April 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

JEPS Bulletin

The Official Blog of the Journal of European Psychology Students

Writing a Systematic Literature Review

Investigating concepts associated with psychology requires an indefinite amount of reading. Hence, good literature reviews are an inevitably needed part of providing the modern scientists with a broad spectrum of knowledge. In order to help, this blog post will introduce you to the basics of literature reviews and explain a specific methodological approach towards writing one, known as the systematic literature review.

Literature review is a term associated with the process of collecting, checking and (re)analysing data from the existing literature with a particular search question in mind. The latter could be for example:

• What are the effects of yoga associated with individual’s subjective well-being?
• Does brief psychotherapy produce beneficial outcomes for individuals diagnosed with agoraphobia?
• What personality traits are most commonly associated with homelessness in the modern literature?

A literature review (a) defines a specific issue, concept, theory, phenomena; (b) compiles published literature on a topic; (c) summarises critical points of current knowledge about the problem and (d) suggests next steps in addressing it.

Literature reviews can be based on all sorts of information found in scientific journals, books, academic dissertations, electronic bibliographic databases and the rest of the Internet.  Electronic databases such as PsycINFO , PubMed , Web of Science could be a good starting point. Some of them, like EBSCOhost , ScienceDirect , SciELO , and ProQuest , provide full-text information, while others provide the users mostly with the abstracts of the material. Besides scientific literature, literature reviews often include the so called gray literature . This refers to the material that is either unpublished or published in non-commercial form (e.g., theses, dissertations, government reports, fact sheets, pre-prints of articles). Excluding it completely from a literature review is inappropriate because the search should be always as complete as possible in order to reduce the risk of publication bias. However, when reviewing the material on for example Google Scholar , Science.gov , Social Science Research Network , or PsycEXTRA it should be kept in mind that such search engines also display the material without peer-review and have therefore less credibility regarding the information they are disclosing.

When performing literature reviews, the use of appropriately selected terminology is essential, since it allows the researchers much clearer communication. In psychology, without some commonly agreed lists of terms, we would all get lost in the variety of concepts and vocabularies that could be applied. A typical recommendation for where to look for such index terms would be ‘ Thesaurus of Psychological Index Terms (2007) ’, which includes nearly 9,000 most commonly cross-referenced terms in psychology. In addition, electronic databases mentioned before sometimes prompt the use of the so-called Boolean operators , simple words such as AND, OR, NOT, or AND NOT. These are used for combining and/or excluding specific terms in your search and sometimes allow to obtain more focused and productive results in the search. Other tools to make search strategy more comprehensive and focused are also truncations – a tool for searching terminologies that have same initial roots (e.g., anxiety and anxious) and wildcards for words with spelling deviations (e.g., man and men). It is worth noting that the databases slightly differ in how they label the index terms and utilize specific search tools in their systems.

Among authors, there is not much coherence about different types of literature reviews but in general, most recognize at least two: traditional and systematic. The main difference between them is situated in the process of collecting and selecting data and the material for the review. Systematic literature review, as the name implies, is the more structured of the two and is thought to be more credible. On the other hand, traditional is thought to heavily depend on the researcher’s decisions regarding the data selection and, consequently, evaluation and results. Systematic protocol of the systematic literature review can be therefore understood as an optional solution for controlling the incomplete and possibly biased reports of traditional reviews.

THE SYSTEMATIC LITERATURE REVIEW

The systematic literature review is a method/process/protocol in which a body of literature is aggregated, reviewed and assessed while utilizing pre-specified and standardized techniques. In other words, to reduce bias, the rationale, the hypothesis, and the methods of data collection are prepared before the review and are used as a guide for performing the process. Just like it is for the traditional literature reviews, the goal is to identify, critically appraise, and summarize the existing evidence concerning a clearly defined problem.

Systematic literature reviews allow us to examine conflicting and/or coincident findings, as well as to identify themes that require further investigation. Furthermore, they include the possibility of evaluating consistency and generalization of the evidence regarding specific scientific questions and are, therefore, also of great practical value within the psychological field. The method is particularly useful to integrate the information of a group of studies investigating the same phenomena and it typically focuses on a very specific empirical question, such as ‘Does the Rational Emotive Therapy intervention benefit the well-being of the patients diagnosed with depression?’.

Systematic literature reviews include all (or most) of the following characteristics:

• Objectives clearly defined a priori;
• Explicit pre-defined criteria for inclusion/exclusion of the literature;
• Predetermined search strategy in the collection of the information and systematic following of the process;
• Predefined characteristic criteria applied to all the sources utilized and clearly presented in the review;
• Systematic evaluation of the quality of the studies included in the review;
• Identification of the excluded sources of literature and justification for excluding them;
• Analysis/synthesis of the information (i.e., comparison of the results, qualitative synthesis of the results, meta-analysis);
• References to the incoherences and the errors found in the selected material.

The process of performing a systematic literature review consists of several stages and can be reported in a form of an original research article with the same name (i.e., systematic literature review):

1: Start by clearly defining the objective of the review or form a structured research question.

Place in the research article: Title, Abstract, Introduction.

Example of the objective: The objective of this literature revision is to systematically review and analyse the current research on the effects of music on the anxiety levels of children in hospital settings.

Example of a structured research question: What are the most important factors associated with the development of PTSD in soldiers?

Tip: In the title, identify that the report is a systematic literature review.

2: Clearly specify the methodology of the review and define eligibility criteria (i.e., study selection criteria that the published material must meet in order to be included or excluded from the study). The search should be extensive.

Place in the research article: Methods.

Examples of inclusion criteria: Publication was an academic and peer-reviewed study. Publication was a study that examined the effects of regular physical exercise intervention on depression and included a control group.

Examples of exclusion criteria: Publication was involving male adults. Studies that also examined non-physical activities as interventions. Studies that were only published in a language other than English.

Tips: The eligibility criteria sometimes fit to be presented in tables.

3: Retrieve eligible literature and thoroughly report your search strategy throughout the process. (Ideally, the selection process is performed by at least two independent investigators.)

Example: The EBSCOhost and PsychInfo electronic databases from 2010 to 2017 were searched. These were chosen because of the psychological focus that encompasses psychosocial effects of emotional abuse in childhood. Search terms were ‘emotional abuse’, ‘childhood’, ‘psychosocial effects’, and ‘psychosocial consequences’.  The EBSCOhost produced 200 results from the search criteria, while PsychInfo produced 467, for a total of 667 articles. […] Articles were rejected if it was determined from the title and the abstract that the study failed to meet the inclusion criteria. Any ambiguities regarding the application of the selection criteria were resolved through discussions between all the researchers involved.

Tip: Sometimes it is nice to represent the selection process in a graphical representation; in the form of a decision tree or a flow diagram (check PRISMA ).

4: Assess the methodological quality of the selected literature whenever possible and exclude the articles with low methodological quality. Keep in mind that the quality of the systematic review depends on the validity and the quality of the studies included in the review.

Examples of the instruments available for evaluating the quality of the studies: PEDro, Jadad scale, the lists of Delphi, OTseeker, Maastricht criteria.

Tip: Present the excluded articles as a part of the selection process mentioned in step 3.

5: Proceed with the so-called characterization of the studies. Decide which data to look for in all the selected studies and present it in a summarized way. If the information is missing in some specific paper, always register it in your reports. (Ideally, the characterization of the studies is performed by at least two independent investigators.)

Place in the research article: Results.

Examples of the information that should and/or could be collected for characterization of the literature: authors, year, sample size, study design, aims and objectives, findings/results, limitations.

Tip: Sometimes results can be presented nicely in a form of a table depicting the main characteristics.

6: Write a synthesis of the results – integrate the results of different studies and  interpret them in a narrative form.

Place in the research article: Interpretation, Conclusions.

Patterns discovered as results should be summarized in a qualitative, narrative form. Modulate one (or more) general arguments for organizing the review. Some trick to help you do this is to choose two or three main information sources (e.g., articles, books, other literature reviews) to explain the results of other studies through a similar way of organization. Connect the information reported by different sources and do not just summarize the results. Find patterns in the results of different studies, identify them, address the theoretical and/or methodological conflicts and try to interpret them. Summarize the principal conclusions and evaluate the current state on the subject by pointing out possible further directions.

CONCLUSIONS

The results emerging from the data that were included in such retrospective studies can lead to a certain level of credibility regarding their conclusions. Actually, systematic literature reviews are thought to be one of our best methods to summarize and synthesize evidence about some specific research question and are often used as the main ‘practice making guidelines’ in many health care disciplines. Therefore, it is no wonder why systematic reviews are gaining popularity among researchers and why journals are moving in this direction as well. This also shows in the development of more and more specific guidelines and checklists for writing systematic literature reviews (see for example PRISMA or Cochrane Handbook for Systematic Reviews of Interventions ). To find examples of systematic literature review articles you can check Cochrane Database of Systematic Reviews , BioMed Central’s Systematic Reviews Journal , and PROSPERO . If you are aware of the concept of ‘registered reports’, it is worth mentioning that submitting with PROSPERO provides you with the option of publishing the latter as well. I suggest that you go through the list of useful resources provided below and hopefully, you can get enough information about anything related that remained unanswered. Now, I encourage you to try to be a little more to be systematic whenever researching some topic, to try to write a systematic literature review yourself and to maybe even consider submitting it to JEPS .

USEFUL RESOURCES

• Cochrane Database of Systematic Reviews : http://www.cochranelibrary.com/cochrane-database-of-systematic-reviews/

EBSCOhost : https://search.ebscohost.com/

Google Scholar : https://scholar.google.com/

PRISMA : http://www.prisma-statement.org/

PROSPERO : https://www.crd.york.ac.uk/prospero/

ProQuest : http://www.proquest.com/

PsycEXTRA : http://www.apa.org/pubs/databases/psycextra/index.aspx :

PsycINFO : http://www.apa.org/pubs/databases/psycinfo/index.aspx

PubMed : https://www.ncbi.nlm.nih.gov/pubmed/

SciELO : http://www.scielo.org/php/index.php?lang=en

Science.gov : https://www.science.gov/

ScienceDirect : http://www.sciencedirect.com/

Scorpus : http://www.scopus.com/freelookup/form/author.uri

Social Science Research Network : https://www.ssrn.com/en/

Systematic Reviews Journal (BIOMED) : https://systematicreviewsjournal.biomedcentral.com/

Web of Science : https://webofknowledge.com/

Other sources

• Sampaio, R. F., & Mancini, M. C. (2007). Systematic review studies: A guide for a careful synthesis of scientific evidence. Brasilian Journal of Physical Therapy, 11 (1), 77-82. doi:http://dx.doi.org/10.1590/S1413-35552
• Tuleya, L. G. (2007). Thesaurus of psychological index terms . Washington, DC: American Psychological Association.

Eva Štrukelj

Eva Štrukelj is currently studying Clinical and Health Psychology at the University of Algarve in Portugal. Her main areas of interest are social psychology and health psychology. Regarding research, she is particularly curious about stigma and with it related topics.

Share this:

Related posts:.

• How to write a good literature review article?
• How to search for literature?
• Editorial perspective on scientific writing: An interview with Dr. Renata Franc
• Peer Review Process – Tips for Early Career Scientists

Easy guide to conducting a systematic review

Affiliations.

• 1 Discipline of Child and Adolescent Health, University of Sydney, Sydney, New South Wales, Australia.
• 2 Department of Nephrology, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• 3 Education Department, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• PMID: 32364273
• DOI: 10.1111/jpc.14853

A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a predefined search strategy to identify and appraise all available published literature on a specific topic. The meticulous nature of the systematic review research methodology differentiates a systematic review from a narrative review (literature review or authoritative review). This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers.

Keywords: research; research design; systematic review.

© 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

Publication types

• Systematic Review
• Research Design*

* Psychology: Literature Reviews

• Psychology Resources
• Literature Reviews
• Research Methods
• Tests and Measures
• Data and Analysis
• FYS 427 - Ecopsychology This link opens in a new window
• EDPY 262: Community-Based Research Methods This link opens in a new window
• PSYC 314: History of Psychology This link opens in a new window

Journals Publishing Literature Reviews

While many scholarly journals may publish literature reviews, some psychology journals are exclusively or mainly devoted to publishing review articles.  When researching a topic, a relevant literature review is an excellent entry point for understanding concepts and the current state of research.

• Annual Review of Psychology
• Annual Review of Clinical Psychology
• Annual Review of Neuroscience
• Clinical Psychology Review
• Current Directions in Psychological Science
• Nature Reviews Neuroscience
• Perspectives on Psychological Science
• Psychological Bulletin
• Psychonomic Bulletin and Review
• Social and Personality Psychology Compass

Writing a Literature Review

Literature reviews analyze and evaluate previously published material on a topic. When they appear in journals, they are referred to as review articles. A systematic review is a kind of review that uses strict methods for identifying and synthesizing previous studies.

The literature review process. From Diana Ridley, The Literature Review: A Step-by-Step Guide for Students (2008), p. 81

For more information about writing a literature review as a senior thesis, see the Bates Psychology Department's Senior Thesis Proposal Guidelines page . Other resources include:

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews (PDF) . Review of General Psychology , 1, 311–320.

Bem, D. J. (1995). Writing a review article for Psychological Bulletin . Psychological Bulletin , 118(2), 172. doi:10.1037/0033-2909.118.2.172

University of Washington Psychology Writing Center.  Writing a Psychology Literature Review (PDF) .

Finding Literature Reviews in PsycINFO

PsycINFO is the premier research tool in psychology. One of its handiest features searching for articles by research methodology, including literature review. From the Advanced Search page, scroll down to the Methodology box in the lower right, and select Literature Review and Systematic Review. Then search the keywords related to your topic.

• << Previous: Psychology Resources
• Next: Research Methods >>
• Updated: Jan 4, 2024 1:51 PM
• URL: https://libguides.bates.edu/psychology

PSYC 321--Social Psychology: Literature Review

• Getting Started
• Tests and Measurements
• Literature Review
• Citing Sources--APA Style

Methods for Synthesizing Qualitative Reviews

Ruth Garside, PhD, Senior Lecturer in Evidence Synthesis at the Medical School, University of Exeter

Check out PRISMA to guide your review: especially the checklist for your review and the flow diagram to develop your protocol.

The PRISMA Statement:

Anybody writing a systematic literature review should be familiar with the  PRISMA statement . The PRISMA Statement is a document that consists of a 27-item  checklist  and a  flow diagram  and aims to guide authors on how to develop a systematic review protocol and what to include when writing the review.

A protocol ideally includes the following:

Databases to be searched and additional sources (particularly for grey literature)

Keywords to be used in the search strategy

Limits applied to the search.

Screening process

Data to be extracted

Summary of data to be reported

From Majumbder, K. (2015). A young researcher's guide to a systematic review. Editage Insights. Retrieved from   https://www.editage.com/insights/a-young-researchers-guide-to-a-systematic-review#

Systematic Literature Review

Here are a couple of articles found in Sage Research Methods Online which give good definitions of what a Systematic Literature Review is and how to do one:

Dempster, M. (2003). Systematic review . In Robert L. Miller, & John D. Brewer

      (Eds.), The A-Z of Social Research. (pp. 312-317). London, England: SAGE

      Publications, Ltd. doi: http://dx.doi.org/10.4135/9780857020024.n110

​Crisp, B.R. (2015). Systematic reviews: a social work perspective . Australian

      Social Work, 68 (3): 284-295. http://dx.doi.org/10.1080/0312407X.2015.102426

Schick-Makaroff, K., MacDonald, M. Plummer, M., Burgess, J., & Neander, W. (2016).

      What Synthesis Methodology Should I Use? A Review and Analysis of Approaches to

       Research Synthesis .  AIMS Public Health, 3 (1). 172-215.

      doi: 10.3934/publichealth.2016.1.172

       http://dspace.library.uvic.ca:8080/handle/1828/7464

Inclusion/Exclusion Criteria

• Veale, T.Search concept tools. Retrieved from //medhealth.leeds.ac.uk/info/639/information_specialists/1500/search_concept_tools Describes various structures for developing criteria: PICO, PICOS, SPIDER, SPICE, etc.

Support for Systematic Reviews

• Systematic Review Search Strategies Worksheet Organize your review by topic, database, search string, and criteria
• Evaluation of Sources Questions to ask of primary source articles (both qualitative and quantitative) when evaluating their quality
• Software for Organizing Systematic Reviews From Columbia University Medical Center's Library
• Evidence-Based Practice
• Meta-Ethnography
• Qualitative Evidence Synthesis

Examples of Systematic Reviews

Prospero: International Prospective Register of Systematic Reviews

This web site collects systematic reviews in process.  By reviewing them, you can see what is included in a systematic review.

Campbell Systematic Reviews

This Monograph series is an open access collection of peer-reviewed systematic reviews.  "Campbell systematic reviews follow structured guidelines and standards for summarizing the international research evidence on the effects of interventions in crime and justice, education, international development, and social welfare." Registration and protocols are available from the Campbell Collaboration Library of Systematic Reviews .

• << Previous: Tests and Measurements
• Next: Citing Sources--APA Style >>
• Last Updated: Jan 18, 2024 3:14 PM
• URL: https://libguides.stthomas.edu/psyc321

© 2023 University of St. Thomas, Minnesota

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 02 April 2024

How do we study misogyny in the digital age? A systematic literature review using a computational linguistic approach

• Lara Fontanella   ORCID: orcid.org/0000-0002-5441-0035 1 ,
• Berta Chulvi 2 , 3 ,
• Elisa Ignazzi 4 ,
• Annalina Sarra 5 &
• Alice Tontodimamma 1  

Humanities and Social Sciences Communications volume  11 , Article number:  478 ( 2024 ) Cite this article

Metrics details

• Cultural and media studies

Nowadays, despite centuries of striving for equality, women still face higher levels of discrimination compared to men in nearly every aspect of life. Recently, this systemic inequality has manifested in cyberspace through the proliferation of abusive content that is even more aggressive than what one would expect in the 21st century. Various research disciplines are now attempting to characterise this new manifestation of misogyny. The endeavour to comprehend this phenomenon has resulted in a significant increase in publications from several fields, including Social Sciences, Arts and Humanities, Psychology, and Computer Science. This paper presents a systematic review of multidisciplinary research on misogyny from the years 1990 to 2022, encompassing a total of 2830 articles retrieved from the Scopus database as of December 31, 2022. The literature is thoroughly analysed using three approaches: bibliometric analysis, topic detection, and qualitative analysis of the documents. The findings suggest that the analysis of online misogyny has been the primary driver behind the exponential growth in publications in this field. Additionally, the results of the topic analysis and topic interaction reveal a limited connection between the areas of knowledge that are necessary to fully grasp this complex phenomenon.

Similar content being viewed by others

The old-new epistemology of digital journalism: how algorithms and filter bubbles are (re)creating modern metanarratives

Luca Serafini

The dark web privacy dilemma: linguistic diversity, talkativeness, and user engagement on the cryptomarket forums

Zhicong Chen, Xiang Meng & Cheng-Jun Wang

Community and authority in ROAR Magazine

Introduction.

Nowadays, regardless of centuries of fighting for equality, women continue to face a disproportionate amount of discrimination compared to men across various contexts. Women and girls encounter prejudice, sexist attitudes, open discrimination, and violence throughout their lives, while the extent of these experiences varies by location, identity, and culture. Disgust, intolerance, or entrenched prejudice, serving to legitimise women’s oppression, persist even in countries often alleged to be post-patriarchal, like the United States, Australia, and the United Kingdom (Manne, 2017 ). The all-pervasive form of hostility and aversion against women and girls is referred to as misogyny, a term derived from the Ancient Greek word “mĩsoguniã”, which means hatred of women. According to Allen ( 2021 ), misogyny has a disputed definition. Some authors offer a definition of misogyny that, in some respects, overlaps with the concept of sexism. For example, Code ( 2000 ) defines misogyny as any of the following acts or feelings: sexual and physical violence against women, exclusion of women, promotion of patriarchy, belittlement, and marginalisation of women. In this approach, the promotion of patriarchy, broadly conceptualised as a system or systems producing and reproducing gendered and intersectional inequalities, is clearly the spread of a sexist mentality. Here, sexism is linked to the acceptance of sex-role stereotypes and can manifest at various levels: individual, organisational, institutional, and cultural (VandenBos, 2015 ). In the same line of reasoning, Jukes ( 1993 ) states that misogyny can be obvious and explicit at times, but it can also be subtle and insidious. However, the subtle expression of misogyny is more linked to sexist attitudes than to the expression of hate. Other authors, such as Manne ( 2017 ), set out a clear distinction between sexism and misogyny. In her book, “Down Girl: The Logic of Misogyny”, Kate Manne ( 2017 ) describes misogyny as the patriarchal order’s “law enforcement” branch, which rewards “good” women who adhere to social norms while punishing those who disobey. Sexism, on the other hand, is viewed as the “justificatory” branch, which rationalises and justifies male dominance through beliefs, theories, stereotypes, and cultural narratives that portray women as naturally inferior. This conceptual debate is due to two reasons. First, the fact that misogyny is strictly linked to the concepts of patriarchy and sexism, and second, the evidence that our societies are facing new ways of conveying misogynistic content in the form of open denigration of women.

Focusing on the link between concepts that describe women’s discrimination, it is evident that the powerful dynamics of a patriarchal society contribute to the development of a sexist culture, and this leads to the oppression of women both in their personal lives and within societal institutions (Millet, 1970 ). Additionally, hostile and benevolent sexism (Glick and Fiske, 1997 ) functions to preserve patriarchy and conventional gender norms. Benevolent sexism manifests through subjectively positive attitudes towards women in traditional roles, encompassing protective paternalism, idealisation, and a desire for intimacy. On the other hand, hostile sexism is expressed in a blatant and resentful way toward women who violate traditional roles and includes the negative equivalents of each dimension of benevolent sexism: dominant paternalism, derogatory beliefs, and heterosexual hostility. The aforementioned patriarchal culture legitimises openly misogynistic expressions, which represent the most extreme manifestation of aggression against women.

In this complex dynamic, studies from different disciplines tend to use different terminology when examining hostility towards women. Specifically, research in psychology is more inclined to use terms related to sexism, especially in distinguishing between hostile and benevolent sexism, and the notion of patriarchy is extensively examined in social science, particularly in sociological studies. The concept of misogyny is more commonly used in communication studies and computational science. The findings reported in the Supplementary Material provide evidence of the emphasis of different disciplines on different concepts.

Regarding the emergence of new ways of transmitting misogynistic content, the rise of interactive social media has been widely considered (Moloney and Love, 2018 ; Rubio Martìn and Gordo Lòpez, 2021 ; Tranchese and Sugiura, 2021 ).

Misogyny on the internet is not a new phenomenon. Indeed, legislation pertaining to women’s online safety dates back to the Beijing Declaration in 1995. However, it was not until the events of Gamergate Footnote 1 (Massanari, 2020 ) in August 2014 that the mainstream media and academic research took notice. In fact, in the gaming community, 2014 saw the emergence of the controversy and online movement known as “Gamergate”. It started out as a reaction to questions about ethics in video game journalism, but it soon turned into a harassment campaign directed at female journalists. The movement brought attention to misogyny, sexism, and the need for diversity in the gaming industry.

With the development of social networks, the historical aversion to women has become articulated through new modes of communication and social interaction. While digital spaces have amplified female voices, online platforms have notoriously facilitated the spread of misogynistic content: women’s systematic inequality and discrimination have been replicated in cyberspace in the form of abusive content much more aggressive than we would have expected in the 21st century (Bates, 2021 ). The online realm provides ample opportunities for misogyny to be linguistically expressed in various ways, ranging from subtle forms such as social exclusion and discrimination to more severe forms like sexual objectification and violent threats (Anzovino et al., 2018 ). Studies examining online misogynistic discourse have employed different terminology, such as “gender cyber hatred” (Jane, 2017 ), “cyber harassment” (Citron, 2014 ), “technological violence” (Ostini and Hopkins, 2015 ), “gender trolling” (Mantilla, 2013 ), “e-bile”, and “gender hate speech” (Jane, 2015 ). Other scholars (see, for instance, Ging and Siapera, 2018 ) chose to use a broader definition of misogyny which almost always results in some form of harm, either directly, in the form of psychological, professional, or physical harm, or indirectly, making the internet a less equal, less safe, or less inclusive space for women and girls.

Our study aims to investigate the current state of research on misogyny. For this purpose, we focus on the scientific literature on this subject during the period between 1990 and 2022. To the best of our knowledge, our study is the first systematic review on misogyny which combines three approaches: bibliometrics, topic detection, and qualitative analysis of the documents.

For the bibliometric research, we first analyse the existing literature extracted from the Scopus database within the misogyny research field by exploiting bibliometric tools. Bibliometric analysis provides a systematic, transparent, and replicable manner to investigate extant literature in a given field and discover the progress of disciplinary research from a macro perspective, supporting future research directions. Using bibliometric methods, we explore the main lines of research in the scientific literature on misogyny and offer a summary of the research activity in terms of the volume of work and evolution over time, as well as in terms of the social, intellectual and conceptual structures of this research area.

Although bibliometric tools provide a broad overview of current research, they cannot deliver detailed insights into studies in the literature based on semantic content analysis. In order to conduct an in-depth semantic analysis, it is necessary to supplement bibliometric methods with text-mining techniques (Hu et al., 2014 ). In accordance, our work employs topic analysis based on the Latent Dirichlet Allocation method (LDA; Blei et al., 2003 ) in order to identify the most prevalent latent themes in misogyny literature. LDA is gaining popularity among scholars in diverse fields (Alghamdi and Alfalqi, 2015 ). Two important findings emerge from a topic model: a list of topics (i.e., clusters of words that appear frequently together) and a list of documents that are strongly associated with each topic. As a result, this method offers a probabilistic quantification of relevance for both the identification of topics and the classification of documents, making it useful for locating interpretable topics with semantic meaning and assigning these topics to literature documents (Tontodimamma et al., 2021 ). According to Suominen and Toivanen ( 2016 ), the main innovation of topic modelling in categorising scientific knowledge is that it essentially eliminates the need to fit knowledge that is brand-new to the world into definitions that are already well-established.

Finally, we complement the study with a qualitative analysis aimed to discover the sociological perspective of the literature on online misogyny, on the one hand, and the computational aspects, on the other hand.

Bibliometric analyses

Bibliographic dataset.

For the analysis, we use a bibliometric dataset covering the period 1990–2022, retrieved from the Scopus database on 31 December 2022. Since we focus on the broad spectrum of scientific research on misogyny, the bibliographic dataset was extracted by looking for publications containing terms related to the generic query “misogyn*” in the content of the title, abstract, and keywords. All types of publications were included in the search, and 2830 documents were retrieved. The top publication fields include Social Sciences, Arts and Humanities, Psychology, and Computer Science.

Information about document distribution by research field is given in the Supplementary Material , along with the document distribution by source and the ranking of the most productive countries and authors.

Research activity

The evolution over time of the number of published documents shows remarkable growth (see Fig. 1 ). We found out that the number of published documents has increased dramatically over time. Since 1992, it has been possible to distinguish two distinct phases. A gradual increase in publications occurred during the first phase, which lasted from 1990 to 2010. The second phase, from 2010 to 2022, has a higher growth rate, indicating increased interest. This finding aligns with the three-stage development theory (Price, 1963 ) of productivity on a particular subject. Small increments in the scientific literature are documented during the precursor period when some scholars begin publishing research on a new topic. The number of papers increases exponentially in the second phase as the topic expands and draws a growing number of scientists, as many facets of the subject remain unexplored. Finally, in the third phase, the curve aspect shifts from exponential to logistic, testifying to a stabilisation in production and a consolidation of the body of knowledge.

Number of publications on misogyny per year: observed and expected temporal evolution according to exponential growth.

To verify the rapid increase, we fit an exponential growth curve to the data. The yearly rate of change in this model is 13.1%, demonstrating how research on misogyny might be cast in the second phase of development: although more research is being released, there is still space for improvement in many areas.

It is noteworthy to highlight that, as shown in the Supplementary Material , the research on misogyny from 1990 to 2002 follows a similar trend as sexism and has a slightly higher yearly growth rate compared to patriarchy. However, when considering only the five years prior to 2022, a more noticeable rise in the volume of published research on misogyny becomes evident, with a twofold increase in the number of published documents.

Social Structure of research on misogyny: collaboration network

To capture the essential characteristics of the misogyny research field, with a specific emphasis on collaborative efforts among different authors, we construct the authors’ collaboration network. We used the Bibliometrix R package, for performing network analysis and visualisation (Aria and Cuccurullo, 2017 ). Within the collaboration network, researchers act as nodes, and the connections between them (edges) represent co-authorships on articles. The node size is indicative of the authors’ productivity, measured in terms of the number of manuscripts authored or co-authored. The edges are weighted according to the frequency of co-authorship. Figure 2 visually illustrates the collaboration network among authors, highlighting the most significant cliques, each distinguished by different colours. The term “clique” is commonly employed to identify highly interconnected groups of elements, such as nodes or vertices, within a network. In our context, a “clique” signifies a group of authors who closely and frequently collaborate with one another compared to their counterparts, thereby creating a densely interconnected structure within the network. The most central scholars, with the highest number of connections, are Elisabetta Fersini, Paolo Rosso, Bilal Ghanem and Viviana Patti, who are also among the most proficient authors in the field of research on misogyny, as shown in the Supplementary Material . The noteworthy aspect is that the densest subgraphs link authors whose research falls under the computer science category.

Authors’ collaboration network.

Intellectual Structure of research on misogyny: citation analysis

The top five documents with the highest number of citations are: “Gamergate and The Fappening: How Reddit’s algorithm, governance, and culture support toxic technocultures” (Massanari, 2017 ), “Down girl: The logic of misogyny” (Manne, 2017 ), “Attitudinal antecedents of rape myth acceptance: A theoretical and empirical re-examination” (Lonsway and Fitzgerald, 1995 ), “Post-postfeminism?: new feminist visibilities in postfeminist times” (Gill, 2016 ) and “Beauty and Misogyny Harmful Cultural Practices in the West” (Jeffreys, 2005 ). These works investigate misogyny from various angles.

Manne’s book explores the logic of misogyny, which “primarily targets women because they are women in a man’s world ” (Manne, 2017 , p. 64). Manne argues that misogyny still exists in alleged post-patriarchal cultures and has taken different forms since legal equality, requiring women to be moral “givers” and validating a sense of entitlement among privileged men. Misogyny often takes the form of taking from women what they supposedly owe men and preventing women from competing for positions of power and authority traditionally held by men. In addition, Manne examines various examples of rape culture, including online harassment.

Considering attitudes toward sexual violence, Lonsway and Fitzgerald investigate the relationship between misogyny and rape myth acceptance. Here, rape myths can be defined as “attitudes and beliefs that are generally false but are widely and persistently held, and that serve to deny and justify male sexual aggression against women” (Lonsway and Fitzgerald, 1994 , pag. 134).

From a feminist perspective, Jeffreys argues that some Western beauty practices (e.g., makeup, high-heeled shoes, breast implants) should be included in the United Nations’ definition of harmful traditional/cultural practices due to the damaging effects they have on women’s health, the creation of sexual difference, and the enforcement of female deference. Gill’s article contends that it is crucial to examine how the media portrays feminism and to delve into the complexities of a cultural moment that seems to be characterised by a range of feminist ideologies (both contemporary and traditional), as well as a resurgence of anti-feminist attitudes and prevalent misogyny.

Massanari’s research centres on online misogyny and is based on a long-term participant-observation and ethnographic study of Reddit’s culture and community. The research specifically focuses on the #Gamergate and The Fappening cases. The Fappening involved the illegal distribution of nude photos of celebrities via anonymous image-board 4chan and Reddit, while #Gamergate was ostensibly about ethics in gaming journalism but became a campaign of harassment against female and minority game developers, journalists, and critics. The study highlights how Reddit’s design, algorithm, and platform politics implicitly support toxic technocultures, providing a fertile ground for anti-feminist and misogynistic movements to flourish. Online misogyny is also discussed in the papers with the highest number of local citations (i.e., citations from other documents within our bibliographic dataset): “#MasculinitySoFragile: Culture, structure, and networked misogyny” (Banet-Weiser and Miltner, 2016 ), “Back to the kitchen, cunt: Speaking the unspeakable about online misogyny” (Jane, 2014 ), and “Drinking male tears: language, the Manosphere, and networked harassment” (Marwick and Caplan, 2018 ).

Conceptual Structure of research on misogyny

To understand the conceptual structure of the research on misogyny, we initially performed an exploratory analysis of the textual content of the keywords chosen by the authors.

Figure 3 shows the most used keywords, after removing the term “misogyny”. Besides the extensive terms gender, feminism and sexism, we find keywords related to the phenomenon of violence against women, to the emerging theme of the Manosphere and to the classical theme of patriarchy and hegemonic masculinity. It is also worth noting the presence of several keywords strictly linked to the online diffusion of misogynistic content.

Most used keywords.

To deepen the analysis, a conceptual structure map (see Fig. 4 ) of the literature on misogyny was created using the Bibliometrix R package (Aria and Cuccurullo, 2017 ), which enables performing multiple correspondence analysis (MCA, Greenacre and Blasius, 2006 ) and hierarchical clustering. MCA, in particular, allows the generation of a low-dimensional Euclidean representation of the original data matrix by performing a homogeneity analysis of the “documents by keywords” indicator matrix, which is constructed by taking into account a dummy variable for each keyword. The words are plotted on a two-dimensional plane, where closer words have a more consistent distribution across the documents.

Conceptual map of research on misogyny.

The two dimensions of the maps that emerged from the MCA can be interpreted as follows. The first dimension separates keywords emphasising the problem of misogyny in general and on social media platforms (on the right) from those related to the methodological aspects of the automatic detection of misogynistic language (on the left). The second dimension separates keywords emphasising the problem of misogyny from a general point of view (on the upper) from those related to the Manosphere and Incels ( involuntary celibates ) and their presence on the Reddit platform (on the bottom). Figure 4 also displays the results obtained through a hierarchical cluster analysis carried out adopting the method of average linkage on the factorial coordinates obtained through MCA. Five clusters emerge from the conceptual structure map. The orange cluster refers to publications related to the automatic detection of misogynistic content through machine learning and deep learning techniques. The green cluster displays the connection between misogyny and hate speech and the exploitation of Natural Language Processing (NLP) methodologies to investigate these phenomena. The blue cluster refers to the intersectionality of research on misogyny. The red cluster is strictly linked to studies of the presence of misogynistic content on social media. Finally, the purple cluster is related to publications dealing with the topics of the Manosphere and the Incel phenomenon.

Research themes in misogyny literature

A topic modelling approach is exploited to investigate the textual content of title, abstract and authors’ keywords to give extra insight into multiple latent themes emerging from the literature on misogyny. To reveal the themes, research interests and trends of studies in the existing misogyny literature, we rely on the LDA model.

Topic analysis: LDA model

LDA is an unsupervised machine-learning-based algorithm allowing to discovery of latent (unobserved) “topics” in large unstructured text datasets (Blei et al., 2003 ). Previous research applied LDA to bibliometrics as an efficient tool for understanding a field’s rich underlying topical structure (see, among others, Suominen and Toivanen, 2016 , Tontodimamma et al., 2021 ). The idea behind LDA is that documents contain multiple topics, and each topic is represented as a probability distribution over terms in a fixed vocabulary, with different topics represented by different probabilities of words in the vocabulary. LDA generative process specifies a joint distribution of hidden and observed variables. The algorithm aims to estimate the posterior distribution of the hidden variables given the observed data, but exact inference is intractable, requiring approximate inference algorithms like sampling-based and variational algorithms (Blei et al., 2003 ; Steyvers and Griffiths, 2006 ). To employ LDA, the user needs to specify the number of latent topics in the corpus and two hyperparameters that control how documents and words contribute to topics. A detailed explanation of the LDA algorithm can be found in the studies by Blei ( 2012 ) and Steyvers and Griffiths ( 2006 ).

In our analysis, we use LDA to model a corpus with each document consisting of the publication title, abstract, and keywords. LDA analysis was performed through the Fitlda Matlab routine, available in the Text Analytics Toolbox (MATLAB, 2022 ).

Topic interpretation

The themes generated by LDA are hidden variables that require proper interpretation, typically done by examining the top keywords associated with each topic (Steyvers and Griffiths, 2006 ). To this end, Figs. 5 and 6 show the most important words for each topic, with importance determined by normalising the posterior word probabilities for each topic by the geometric mean of the posterior probabilities for the word across all topics. The topics are ranked based on their estimated likelihood of being observed in the entire dataset. Section 2 of the Supplementary Material contains the list of the most significant terms and their relevance measurements. The twelve selected topics address crucial areas of research on misogyny and can be summarised as follows.

Word clouds for topics 1–6.

Word clouds for topics 7–12.

Topic 1 revolves around a comprehensive discussion on the feminist perspective of the misogyny phenomenon and addresses the root causes of misogyny and gender-based discrimination. The primary focus is on patriarchal male gender privilege and its role in perpetuating misogyny. This topic covers a range of issues related to gender inequality, such as the leadership gap between men and women, women’s rights, and the intersection of misogyny with other forms of oppression.

Topic 2 focuses on how misogyny is expressed in literary works from the early and medieval periods to the modern era. Overall, this topic highlights the role of novels, prose, tales, and fiction in shaping societal attitudes and beliefs about gender and how this has influenced the treatment of women throughout history.

Along similar lines, Topic 3 centres on the study of misogyny in relation to the representation of women in films and on how it influences the portrayal of women on visual media.

Topic 4 is focused on the study of misogyny within the realm of politics and examines how misogyny can be perpetuated within political systems and movements. In particular, the inclusion of terms like “American”, “white”, and “altright” suggests that research included in this topic might focus on the ways in which misogyny is manifested in American politics, particularly within white nationalist and alternative-right movements.

Topic 5 is centred on the study of masculinity and how it relates to misogyny. In particular, the word “hegemonic” suggests that this topic may focus on how dominant forms of masculinity reinforce misogyny and gender-based discrimination.

Topic 6 pertains to the research on women’s rights, including reproductive rights, family law, and access to healthcare, particularly within legal and political systems and on how these systems can either promote or hinder gender equality.

The latent theme of Topic 7 seems to refer to a broad subject area that encompasses issues related to education, sexuality, and sexual identity. Additionally, the related terms suggest a focus on the ways in which sexuality is addressed within educational institutions, including schools and universities.

Topic 8 is a subject area that focuses on the study of digital misogyny, which refers to the ways in which sexism and gender-based discrimination are perpetuated through online and digital media platforms.

The set of words linked to Topic 9 clearly indicates studies focusing on the subject of sexual violence and harassment.

Research included in Topic 10 is related to the investigation of misogyny in the context of music and religion.

Topic 11 appears to be focused on the intersection of misogyny and racism, particularly as it relates to the misogynoir phenomenon.

Finally, Topic 12 deals with the identification and classification of online misogyny.

Topic interactions

By modelling each document as a mixture of several topics and each topic as a combination of words, the LDA technique assigns topics to documents. In our analysis, we awarded the top three document-topic probabilities to each document in this study as long as the probabilities are greater than 0.2. We developed a topic relationship network by considering the topic co-occurrence matrix. The topic network is depicted in Fig. 7 , along with node centrality measures. The nodes are coloured according to their degree, and the edges are weighted depending on co-occurrences. The stronger the link, the thicker the line. Edges with weights less than the average number of co-occurrences have been omitted. The investigation of the linkages reveals relationships between research fronts, emphasising the multidisciplinary character of research on misogyny. The highest degrees are associated with the first three topics, which encompass broader themes dealing with the feminist perspectives of patriarchal society (Topic 1) and the representation of women in literary works (Topic 2) and cinema (Topic 3). Moreover, the latter two topics show the strongest interconnection. Lower degrees are associated with more specialistic research fronts related to the presence of misogyny in music and religion, the misogynoir phenomenon, and the recent field of misogyny detection in computational sciences. In particular, the theme of automatic identification of misogynistic content (Topic 12) is only linked to the research dealing with digital misogyny (Topic 8). A high betweenness, measuring the extent to which the node is part of paths that connect an arbitrary pair of nodes in the network, is associated with Topics 5 and 6, dealing with the study of masculinity and how it relates to misogyny and to research on women rights, respectively. These findings suggest that those research areas are more effective and accessible in the network and form the densest bridges with other nodes.

Topic co-occurence network for the publications on misogyny and nodes’ centrality measures.

Topic temporal evolution

The temporal evolution of the scientific productivity for each topic can be captured through Fig. 8 . The temporal trend of most topics agrees with exponential growth. However, looking at Topic 2, related to studies of misogyny in literary works, we notice how the number of publications in the last period falls below the number expected according to the exponential law. Conversely, the number of published documents for Topics 8 and 12 shows a sudden rise starting from 2018. This trend testifies to the increasing interest in the study of online misogyny and the related techniques for automatic detection and identification. A relatively more contained rise in the size of publications is recorded for Topics 10 related to the investigation of misogyny in the context of music and religions.

Number of publications on misogyny for each topic: observed and expected distributions according to exponential growth.

The appearance and development of new fields of interest and innovative ideas in the research activity on misogyny are confirmed by the heatmaps provided in the Supplementary Material , which show the number of documents, by years, assigned to the identified topics.

Sociological research on online misogyny

To improve our comprehension of the ongoing research on the online dissemination of misogynistic content, we utilised a more specific selection query in our search of the original set of documents, which targeted terms associated with the online environment. We limited our search to articles published in journals categorised under the Social Science subject area. After analysing 277 articles, we identified 187 that were suitable for our study.

Among these documents, four articles provide a review of the literature on online misogyny from different perspectives. Moloney and Love ( 2018 ) review the way online misogyny is conceptualised in the social scientific literature within feminist media studies. The authors identify four different terms that are used to describe different types of online misogyny: online sexual harassment, gendertrolling, e-bile, and disciplinary rhetoric. They also examine the sociological perspective and introduce the concept of “virtual manhood acts” (VMAs), which is situated within the broader context of critical gender theory. VMAs are examples of technologically facilitated misogyny that occur in online social spaces: textual and visual cues are exploited to signal a masculine self, enforce traditional gender norms, oppress women, and restrict men to predefined gender roles. Bosch and Gil-Juarez ( 2021 ) conducted both a systematic review of 33 articles found in Web of Science and a traditional review of academic, institutional, and feminist-activist publications. Their findings show that the majority of aggressors in online gender-based violence are cis-hetero-patriarchal men, who are mostly known to the victims and are often partners or ex-partners. The types of violence range from sexual insults and threats to sexual and high-tech violence. Rubio Martìn and Gordo Lòpez ( 2021 ) provide an overview of the most recent academic literature within the feminist technosocial literature, specifically related to sexual and gender-based violence in digital environments. In addition to discussing the contemporary antecedents of this perspective and presenting current positions and the most representative studies on topics related to online misogyny, the authors demonstrate the potential of the feminist technosocial approach for analysing digital environments and their designs. The main conclusion drawn is that both the values of a misogynistic culture and the possibilities for its reproduction and dissemination are embedded in the design and architecture of digital platforms. The article also highlights the increasing relevance of hybrid realities that result from the synergies between the physical and digital realms, as they enable amplified discourses and actions of online misogyny. Faith ( 2022 ) investigates how gender, technology, and development are interconnected by analysing various works from different fields, including feminist technology studies, gender and development, feminist criminology, and ICT for development. The study also draws data from sources such as civil society, news reports, and international organisations, like the UN, to examine online violence. The author argues for a critical research approach to better understand the complex and opaque power dynamics that shape the digital economy and how it affects gender and development goals.

The articles on online misogyny, which were found in the Social Science category, underwent a manual annotation process to extract various pieces of information. Regarding the different methodologies and techniques used to investigate online misogyny, our findings indicate that discourse analysis and content analysis are the primary methodologies employed in social science literature. Several studies utilise in-depth interviews and surveys to examine the individuals targeted by and responsible for online misogyny. Additionally, digital ethnography, corpus linguistics, and network analysis are also employed. The most analysed social media platforms include Twitter, Reddit and Facebook. Further details on the methodological approaches and the social networks are provided in the Supplementary Material . The subsequent sections delve into details regarding target victims, misogynistic groups, and potential measures to counteract online misogyny.

Targets of online misogyny

Scholars studying online misogyny have identified various target groups that are particularly vulnerable to misogynistic content. These groups include female politicians, journalists, celebrities, influencers, musicians, gamers and developers, YouTubers, university students, and women who have been sexually assaulted. By focusing on specific target groups, research helps in achieving a more nuanced understanding of the ways in which online misogyny manifests and the specific harms that it causes.

Studies on online misogyny directed towards female politicians have concentrated on analysing the experiences of women from various countries, examining the types of misogynistic content directed towards them and the platforms on which it is disseminated. Silva-Paredes and Ibarra Herrera ( 2022 ), using a corpus-based critical discourse analysis, explore abuse received by a Chilean right-wing female politician. Phipps and Montgomery ( 2022 ) conducted an investigation into the portrayal of Nancy Pelosi as the monstrous feminine in the deeply misogynistic attack advertisements of Donald Trump’s 2020 presidential re-election campaign. In light of the prevalent misogynistic and anti-feminist depictions of Senator Hillary Clinton across all types of media, Ritchie ( 2013 ) examines how online media continues to have the power to create harmful representations of female politicians and the consequences for the political campaigns of women and for the democratic process as a whole. Focusing on Canadian politicians, Wagner ( 2022 ) discusses how online harassment is a gendered phenomenon. The study, drawing upon interviews with 101 people from diverse genders, racial/ethnic identities, sexual orientations, and partisan affiliations, shows that women are more aware of online harassment than men and how it succeeds in making women feel they are in a hostile political environment. Saluja and Thilaka ( 2021 ), analysing the Twitter discourse referring to three well-known female politicians in India, reveal similar findings, emphasising how female politicians are subjected to a different and distinct pattern of reception compared to their male counterparts. Instances of misogynistic or sexist hate speech and abusive language against female politicians in Japan are investigated in Fuchs and Schäfer ( 2021 ).

The research conducted by Chen et al. ( 2020 ) through in-depth interviews with 75 female journalists from Germany, India, Taiwan, the United Kingdom, and the USA revealed that those journalists frequently encounter online gendered harassment. The harassment, which includes sexist comments that criticise, attack, marginalise, stereotype, or threaten them based on their gender or sexuality, has led to some female journalists being subjected to misogynistic attacks and even threats of sexual violence. The study suggests that this kind of harassment limits their level of interaction with their audience without being attacked or sexually undermined.

By examining the findings of the qualitative in-depth interview of 48 female journalists, Similar findings are reported by Koirala ( 2020 ), whose study, based on the qualitative in-depth interview of 48 female journalists in Nepal, highlights how some of them tolerate harassment by being ‘strong like a man’, while many avoid social media platforms to keep free of such abusive behaviour. Along the same lines, Rego ( 2018 ) analyses Twitter conversations with Indian journalists and argues that social media platforms constitute convenient havens of harassment against assertive women.

Ghaffari ( 2022 ), analysing user-generated comments on the Instagram profile of a female American celebrity, shows how women are required to suppress their feelings and limit their authentic online presentation to maintain the outward countenance that matches the stereotypical gender roles in audiences’ state of mind. The research conducted by Döring and Mohseni ( 2019 ) supports these findings, focusing on video producers on YouTube. Their study found that female video producers are more likely to receive negative comments compared to male producers, but only if they display their sexuality or address feminist topics. However, if they conform to traditional gender role expectations, they do not experience this kind of negative feedback.

The emergence of female gamers in video game communities has led to a rise in misogynistic attacks against those who challenge the traditional hypermasculine culture of gaming. The 2014 #gamergate incident is a prime example of this, where a group of gamers opposed “Social Justice Warriors” who highlighted discrimination and exclusion in the gaming industry. Female gamers were subject to death threats, rape threats, and doxxing, where their private information was shared online (Tomkinson and Harper, 2015 ). The video gaming community has a long history of gender-based attacks on women, which serve to create a toxic environment for them when making and playing video games. According to Jenson and De Castell ( 2021 ), who approach the subject from a feminist perspective, video games have been predominantly masculine and gendered spaces. Repeated displays of aggression, referred to as “shock and awe”, perpetuate and legitimise gendered hostility. These displays also help to preserve exclusionary media practices designed to maintain the status quo.

The Manosphere

Numerous articles on online misogyny examine the Manosphere, a collection of websites and social media groups that endorse misogynistic beliefs. These networks are not uniform but consist of multiple misogynistic groups with differing perspectives and degrees of violence, which are associated with far-right, homophobic, and racist ideologies (Dickel and Evolvi, 2022 ). Despite their variations, all these groups portray feminism as innately discriminatory and threatening to men (Farci and Righetti, 2019 ). The Manosphere adheres to the beliefs of a ‘gynocentric order’ and the Red Pill ideology, a metaphor derived from the movie The Matrix, in which the protagonist’s eyes are opened to reality upon taking the “red pill”. Although these groups may have distinct beliefs, many members use the term misandry, referring to the hate against men, which has ideological and community-building functions. It reinforces a misogynistic belief system that portrays feminism as a movement that hates men and boys (Marwick and Caplan, 2018 ). The use of misandry caters to both extremist misogynistic subcultures and moderate men’s rights groups. It enables these groups to adopt the language of identity politics, positioning men as the silenced victims of reverse discrimination in all aspects of political, economic, and social life and solidifying their sense of entitlement (Farci and Righetti, 2019 ).

Men’s rights activists employ a personal action frame to construct a plausible but fictional narrative of men’s oppression (Carian, 2022 ). The movement against feminism revolves around advocating for men’s rights while denying that gendered violence exists (Garcìa-Mingo et al., 2022 ). The Manosphere engages in a crucial ideological effort that normalises, trivialises, and legitimises sexual violence against women in various forms (Garcìa-Mingo et al. 2022 ). Some of the primary themes of this ideology are: criticising and verbally abusing women, downplaying or not taking seriously accusations or reports of rape, depicting #MeToo as a feminist plot, portraying men as victims, and advocating for the restoration of patriarchal values (Dickel and Evolvi, 2022 ). Hopton and Langer ( 2022 ), analysing Twitter content to understand how the Manosphere constructs masculinity and femininity, identify three discursive strategies: co-opting discourses of oppression, naming power, and disavowal by disaggregation. These strategies are used to position men as victims, portray women as monstrous others, and re-establish gendered power hierarchies through continuous references to rape in their discourse.

One of the main groups in the Manosphere, the Incels, believes in a hetero-patriarchal racial hierarchy and justifies their lack of sexual activity through ideas rooted in biological determinism and victimisation by women and feminism (Lindsay, 2022 ). Scotto di Carlo’s analysis of Incels (Scotto di Carlo, 2023 ) reveals a conflation of apparently sarcastic metaphors, dark humour, and misogyny to describe women, as well as unique self-representations of forum participants that do not conform to typical ‘us vs them’ identity pattern (van Dijk, 1998 ): instead of highlighting the positive qualities of their in-group, the Incels describe themselves in a derogatory manner, leading to a spiral of self-pity and self-contempt that can foster a sense of brotherhood within the community. These findings stem from a content-discourse analysis of posts from threads specifically discussing women on an incel forum and from the study of nominations and predications of self-representations used in the ‘Introductions’ thread of the same forum. Halpin ( 2022 ), drawing on a qualitative analysis of comments made on a popular Incel discussion board, demonstrates how the group uses its perceived subordinate status to justify their misogyny and legitimise its degradation of women. Conducting an ethnographic content analysis of incel-identified subreddits and using femmephobia as a lens, Menzie ( 2022 ) examines how Incels employ heteropatriarchal conceptions of femininity to devalue women and to describe the social conditions that force them to remain celibate. The study focuses on the symbolic actors constructed by Incels, namely Stacy, who represents the most attractive women, Becky, who represents women of ordinary or moderate attractiveness, and Chad, who represents dominant alpha males. Five themes emerge from the analysis. First, Incels use these symbolic gendered actors to describe a sex deficit most men suffer, implying their own undesirability. Second, Incels’ femmephobia towards hyper-feminine women for not fitting heteropatriarchal requirements is evident in “Stacy”.Third, “Becky” shows a more flexible femmephobia towards women of different appearances who uphold “unrealistic standards” and date men more attractive than themselves or rely on feminism to cope with not attracting the same men as Stacy. Through “Chad”, the fourth topic examines the idea of masculinity, incorporating feelings of jealousy and recognition of victimisation under societal conditions that allow women to exploit men financially or emotionally. Finally, the analysis reveals how Incels prioritise partner display as a symbol of wealth. Along the same lines, Koller and Heritage ( 2020 ) analysed a textual corpus created from threads posted and commented on by Incels. The study examined keywords, word frequencies, and concordance lines to explore the representation of gendered social actors. The findings suggest that Incels position different groups of men in a hierarchy in which conventionally attractive men occupy the top position. Female social actors are not placed in a similar hierarchy. Furthermore, an additional appraisal analysis of the most frequently occurring male and female social actors reveals that men are judged as unable to function, while women are viewed as immoral, dishonest, and capable of causing harm to men.

Chang ( 2022 ), analysing the discourses circulating on a Reddit forum for self-proclaimed Incels, explores the perceptions created by the term “femoid”, a derogatory term generated by Incels to refer to women, constructing them as an abject “monstrous-feminine”, serving a dehumanising function and thus justifying the violence enacted upon them. Tranchese and Sugiura ( 2021 ) focus on the similarities between the language used in pornography and that of Incels, arguing that both are different manifestations of the same misogyny. Their study involves a linguistic analysis that compares a collection of posts from an Incel subreddit community with a reference collection of posts from 688 subreddits covering other subjects. From a different perspective, Byerly ( 2020 ) investigated news media language in the coverage of Incel behaviour associated with sexual aggression. The study employs qualitative textual analysis on a sample of 70 articles obtained using keyword combinations ‘incels and violence’, ‘incels and social media’, and ‘incels and sexism’ from 29 distinct news sources across 6 countries throughout the years 2018 and 2019. Research findings indicate that news stories emphasise the role of social media in helping Incels find each other and form online communities. Additionally, specific social media sites served as locations to amplify misogynistic attitudes and to boast about their murders. Speckhard et al. ( 2021 ) conducted a study that involved gathering information on Incels’ social and personal lives, adherence to incel ideology, opinions on incel-related violence, support for violent actions, and beliefs regarding the classification of Incels as violent extremists. The data was collected through a Google Forms survey that was distributed to active adult members of a prominent Incel forum. The final sample under analysis comprises 272 respondents who self-identify as Incels. The findings demonstrate that while most of them do not advocate violence and are non-violent, those who strongly hold misogynistic beliefs are more likely to endorse violent actions. Participation in Incel online forums, which validate their viewpoints, could also lead to an increase in their misogyny. O’Donnell and Shor ( 2022 ) investigated how misogynistic Incels discuss mass violence committed by their peers. Through qualitative content analysis of comments related to the 2018 Toronto van attack, in which self-declared Incel Alek Minassian drove a van into pedestrians, killing 10 and injuring 16, they found that a large majority of self-proclaimed Incels expressed support for such violence, as well as violence in general. Incels believed that mass violence was a means to achieve four main goals: gaining more attention, seeking revenge, reinforcing traditional masculinity, and bringing about political change.

MGTOW (Men Going Their Own Way), a separatist group within the Manosphere, also promotes a misogynistic agenda. Unlike Men’s Rights Activists and Incels, MGTOWs focus on individualistic and self-empowering actions, encouraging men to lead a self-sufficient life away from women. Jones et al. ( 2020 ), using content and thematic analyses of a corpus of tweets from three of the most active MGTOW users on Twitter, have linked the MGTOW ideology with toxic masculinity, showing that the online harassment it generates is deeply misogynistic and upholds heterosexual and hegemonic masculinity. The authors note that, although misogyny and violence produced by MGTOW are not extreme, the group’s appeals to rational thinking make them appear to be common sense. Wright et al. ( 2020 ) delve deeper into the structural underpinnings and nature of MGTOW debate within their discussion forums, including leadership, moderation, in-group dynamics, and the discursive form of debates, and how this contributes to the propagation of misogyny and different calls to action. The authors conducted a content analysis of comments in the official MGTOW website’s forum and a digital ethnographic approach. Their findings showed that discussions primarily revolve around women and the MGTOW community. When discussing women, users did so in an openly misogynistic way. When discussing MGTOW, conversations sought to define and rationalise it as an ideology, both for individuals and the collective. The authors also note that the communicative form was mainly communitarian, with strong group bonding, ties, and engagement.

Countering online misogyny

Strategies and tactics used by women to cope with and address gender violence online are diverse and sometimes activated simultaneously. Some of these strategies prioritise self-care and protection, while others focus on resistance and challenging such violence. From a self-care perspective, it is crucial to adopt mitigation measures that reduce harm and minimise risks, such as assessing online identities, adopting pseudonyms or collective identities, using masks, strengthening accounts, creating distance, silencing or erasing sensitive content (Bosch and Gil-Juarez, 2021 ). In the research by Chen et al. ( 2020 ), it is shown how female journalists have developed multiple strategies for coping with abuse, including modifying their social media postings, altering their reporting subjects, and utilising technological tools to prevent offensive comments on their public pages.

Merely prioritising self-care is insufficient; an active approach should be taken to resist and transform the current state of online misogyny. This involves engaging in actions that challenge the status quo and strive for meaningful change, with the ultimate goal of repoliticising the internet and social media with, for, and from a feminist perspective (Bosch and Gil-Juarez, 2021 ). From this standpoint, social media platforms can give space to the promotion of gender-based harassment but can also serve as crucial spaces for feminist education and activism and for the formation of a feminist counter-public that directly contests a misogynistic culture (Sills et al. 2016 ). In this perspective, Kurasawa et al. ( 2021 ) discuss a new form of feminist activism called evidentiary activism, which uses evidence of gender-based online violence (GBOV). Evidentiary activism engages with existing formal evidentiary cultures by advocating for legislative and regulatory reforms to address GBOV, promoting platform-based technological solutions, and challenging conventional notions of user privacy and anonymity. In addition, it involves contributing to and embracing informal evidentiary cultures, which use evidence as a tool for cultural and political mobilisation against GBOV. Strategies used include publicising instances of GBOV, highlighting the moral implications of such violence, and fostering feminist digital citizenship. As an example of online feminist activism, Kim ( 2017 ) explored the role of the 2015 hashtag #iamafeminist in promoting feminist identification and activism against misogyny in South Korea. The hashtag persisted for three months by addressing current gender issues and promoting activism both online and offline. The article by Shesterina and Fedosova ( 2021 ) examines the methods used by female bloggers to promote feminist ideas on Instagram. The authors found that while many posts are logically argued, female bloggers often use emotional manipulation and persuasion techniques to promote their ideas. The study identifies both the main topics in support of feminism, such as domestic violence and gender stereotypes, victim blaming, and the most common attitudes that female bloggers challenge in their posts (e.g., “gender roles are determined by nature”, “a woman must obey a man”, “female intelligence is worse than male”, “all women are hysterical”). The authors also describe the lexical means and rhetorical techniques commonly used in female blogs, such as metaphors, allusions, appeals, and rhetorical questions. The language used is generally colloquial, making texts easier to read, but it also includes harsh criticism and increased emotionality compared to traditional journalistic texts.

However, according to Jane ( 2016 ), taking matters into one’s own hands when faced with online harassment may have limited effectiveness and is not a sufficient solution to the problem of gendered cyber-hate. This approach shifts the responsibility from the perpetrators to the targets and the private sphere rather than addressing the broader social issue. The author suggests that a combination of feminist activism efforts, including a revised approach to collectivism, is needed to enact the necessary legislative and corporate changes to combat gendered online hate. The study by Davis and Santillana ( 2019 ) examines the potential and limitations of digital media activism in raising awareness about gender-based harassment using the case study of Las Morras, a Mexico City-based feminist media group. The study demonstrates the paradoxical role of networked digital media as an activist tool. While it rapidly circulated a critique of misogyny, it also attracted negative attention, leading to the group’s eventual demise due to doxing, trolling, and personal threats directed at its members.

Megalians, a cyberfeminist community in South Korea, utilised the technique of “mirroring” to combat online misogyny (Jeong and Lee, 2018 , Moon et al., 2022 , Yang and Lee, 2022 ). This practice involved mimicking the language of misogynistic online communities and reversing the roles of perpetrators and victims. Megalians also used parodies to subvert the humour and power dynamic that men often used to make fun of women. By appropriating and using the language of misogynists, they aimed to strip men of their ability to use misogynistic speech for their own entertainment. This approach also exposed the absurdity and ridiculousness of the misogynistic rhetoric. However, the success of mirroring is not clear-cut. In fact, while Megalians’ voices were heard in society, the strong message and crude language proved divisive and polarising (Kim, 2021 ).

An alternative strategy for addressing misogyny is to use social re-norming and appeal to the empathy of those engaging in harassing behaviour. The goal of re-norming is to challenge cultural attitudes and beliefs that tolerate or encourage violence against women and to promote new standards of behaviour that prioritise respect, equality, and safety for all individuals. One example of this approach is the experiment conducted by Whiley et al. ( 2023 ) on Twitter. Their experiment aimed to inform misogynistic offenders that their sexist language was disapproved of by the majority of people. However, this intervention did not result in a reduction in the number or frequency of sexist Tweets or users, nor did it affect the tone or emotional intensity of subsequent tweets. In contrast, research has demonstrated the efficacy of creative humour, such as that used by the IncelTears subreddit to ridicule Incels, in promoting (dis)affiliative and informative functions (Dynel, 2020 ).

Computational science research on online misogyny

In this section, we focus on documents on misogyny classified by Scopus in the “Computer Science” subject area. A total of 196 documents were found; 30 documents were excluded as they were off-topic. Two surveys were identified in the retrieved documents, which centre on the automated detection of online misogyny. In one survey, Shushkevich and Cardiff ( 2019 ) present an examination of techniques for identifying misogyny in social media through automation. Meanwhile, Sultana et al. ( 2021 ) conducted a systematic literature review of prior research to reveal different aspects of misogyny and sexist humour and to create a codebook for annotation purposes.

Automatic detection of misogyny

Manual classification of the retrieved articles reveals a wealth of valuable information regarding the automatic detection of misogyny. This includes details about the social networks that are being analysed, the primary techniques employed, and the availability of datasets.

In line with research in the social science area (see Section 4), Twitter (with 95 publications) and Reddit (with 46 publications) continue to be the most commonly used sources, even in the area of computational science. The number of studies dealing with Facebook and Instagram is very limited. Researchers frequently prioritise the study of Twitter (now rebranded X) and Reddit above other social media platforms due to their historically liberal provision of Application Programming Interface (API) access. Furthermore, Reddit, which has been described as ’a community of communities’ (Massanari, 2017 , p. 331), has a diverse array of subreddits that cater to different interests, some of which foster misogynistic beliefs. However, the new pricing plans for using the Twitter API, introduced in March 2023, are expected to significantly affect research. A survey conducted by the Coalition for Independent Technology Research Footnote 2 outlines the potential consequences of discontinuing free and affordable API access. These drawbacks include the disruption of research on the dissemination of harmful content. A similar survey on the impact of Reddit’s recent API changes Footnote 3 emphasises how researchers are concerned about interruptions in their research resulting from API modifications. It is worth noting that only one study (Semenzin and Bainotti, 2020 ) reports the results of research on Telegram, which, in fact, has become a widely used platform for the dissemination of abusive and misogynistic content due to its high degree of anonymity and limited content-moderation policies (Guhl and Davey, 2020 ).

The automatic detection of misogyny typically utilises various techniques, with pre-trained deep-learning models and multimodal models being the most commonly employed. Other techniques include machine learning algorithms such as SVM, Naïve Bayes, or Random Forest. Additionally, some documents rely on convolutional neural network models. More details on the published documents employing the different techniques are provided in the Supplementary Material .

Four articles employ the use of lexicons for automatic detection of misogyny. Attanasio and Pastor ( 2020 ) propose misogyny lexicons for automatic misogyny identification in order to improve sentence embedding similarity. Hurtlex (Bassignana et al. 2018 ), which is a lexicon of offensive, aggressive, and hateful words in more than 50 languages, is exploited for misogyny identification in the studies by Chiril et al. ( 2022 ) and Pamungkas et al. ( 2018 ). Kwarteng et al. ( 2022 ) created a specific lexicon around misogynoir.

Taxonomies and guidelines

When releasing annotated datasets, a crucial aspect is to clearly outline the guidelines for categorising misogynistic language. Four articles in the retrieved documents address this issue (Anzovino et al., 2018 , Guest et al., 2021 , Sultana et al., 2021 , Zeinert et al., 2021 ).

Sultana et al. ( 2021 ) proposed eleven categories to classify misogynistic remarks: Discredit (slurring over women with no other larger intention), Stereotyping (description of women’s physical appeal and/or comparisons to narrow standards), Sexual harassment (to physically assert power over women), Threats of violence (intent to physically assert power over women or to intimidate and silence women through threats of violence), Dominance (to preserve male control, protect male interest and exclude women from the conversation), Derailing (to justify abuse, reject male responsibility, and attempt to disrupt the conversation in order to refocus it), Victim blaming (blaming the victims for the problems they are facing), Mixed bias (gender bias might be mixed with other kinds of biases like religious or racial), Sexual objectification (evoke sexual imagery), and Damning (contains prayers to hurt women). Regarding the expression of misogyny using humour, this research proposes eight categories of jokes: Devaluation of personal characteristics, Women’s place in the private sphere, Violence against women, Feminist backlash, Sexual objectification, Excluding and/or objectifying humour, Transphobic Jokes and Cruel or Humiliation. All the categories proposed in Anzovino et al. ( 2018 ) are included in Sultana et al. ( 2021 ). The same occurs with categories proposed by Zeinert et al. ( 2021 ), except for the interesting concept of neosexism. Neosexism is a concept defined in Francine Tougas et al. ( 1999 ), and presents as the belief that women have already achieved equality and that discrimination of women does not exist. Neosexism was the most common form of misogyny present in the dataset of Zeinert et al. ( 2021 ). Guest et al. ( 2021 ) define four categories of misogynistic content: misogynistic pejoratives, descriptions of misogynistic treatment, acts of misogynistic derogation and gendered personal attacks against women.

Evaluation campaigns

A number of the documents on misogyny that fall within the Computer Science subject area were produced in connection with various evaluation campaigns. These campaigns include EVALITA (Evaluation of NLP and Speech Tools for Italian), IberLEF (Iberian Languages Evaluation Forum), SemEval (International Workshop on Semantic Evaluation), and FIRE (Forum for Information Retrieval Evaluation). The EVALITA campaign includes the Automatic Misogyny Identification (AMI) task (Fersini et al. 2018 ). The IberLEF annual campaign features the EXIST task, which is sEXism Identification in Social neTworks (Rodrìguez-Sanchez et al. 2021 ). SemEval has a task called MAMI, which is Multimedia Automatic Misogyny Identification (Fersini et al., 2022 ). Lastly, FIRE includes the Arabic Misogyny Identification (ArMI) task (Mulki and Ghanem, 2022 ).

Thanks to these evaluation campaigns, datasets for automatic misogyny detection in multiple languages are now available. Specifically, the AMI task made available two datasets, in English and Italian, downloaded from Twitter. The EXIST task provided datasets of tweets in both English and Spanish. The dataset released for the MAMI challenge comprises memes that were downloaded from popular social media platforms such as Twitter and Reddit, as well as from websites dedicated to meme creation and sharing. Lastly, the ArMI task provided a dataset of tweets written in Modern Standard Arabic (MSA) and various Arabic dialects.

The bibliometric analysis reveals that research on misogyny has witnessed exponential growth from 2010 to 2022. This growth can be attributed to various areas of research, but one prominent factor contributing to this trend is the increased attention given to the online dissemination of hate towards women. Several findings support this initial conclusion.

Firstly, the analysis indicates that the most productive authors in the field of misogyny research come from the area of computer science. This suggests that experts in this field have been actively investigating and publishing on the topic, further driving the growth of research in this area.

Moreover, examining the topics covered in the analysed documents provides additional evidence for the influence of online misogyny. Topic 8, which is related to digital misogyny, and Topic 12, which focuses on the automatic identification of misogyny in social media, have experienced significantly higher growth compared to the broader field of misogyny research (as depicted in Fig. 8 ). This finding indicates that the study of misogyny in online platforms and the development of methods to detect misogyny in social media have gained considerable attention within the research community.

The major role that online misogyny plays in the development of the area supports the idea that the research seeks to delineate the contours of a new face of misogyny, the latest manifestation of hate towards women which is expressed more crudely and more openly on social networks because they facilitate anonymity and a greater distance from the victims.

Another conclusion drawn from the analysis of the conceptual structure of misogyny research (Fig. 4) and the interactions between topics (Fig. 7 ) is that the research focused on the automatic detection of misogyny in online platforms (Topic 12) exhibits weak connections with other conceptual areas that address different aspects of the phenomenon. This area of research only demonstrates some conceptual relation to the broader study of online misogyny (Topic 8). This presents a significant challenge, considering that qualitative analysis of sociological research emphasises the growing relevance of hybrid realities resulting from the synergies between the physical and digital realms, not just in violence against women but also in specific domains such as politics. Moreover, the lack of relationship between Topic 12, which focuses on the automatic detection of misogyny, and Topic 9, which explores violence against women and the concept of Manosphere (primarily a digital phenomenon), is particularly noteworthy. This suggests that research in the computational science domain may not be adequately addressing the most extreme manifestations of online misogyny. Furthermore, it also indicates that the tools offered by computational linguistics are underutilised in social science-led research.

In general, the absence of stronger connections between certain topics that attract the attention of various disciplines could be seen as a sign of the practical challenges encountered in interdisciplinary research. For instance, Topic 6, which focuses on the study of women’s rights within legal and political systems, exhibits very weak relationships with Topics 8 and 12, despite qualitative sociological research emphasising the need to consider the new dynamics emerging in virtual spaces. Another illustration can be found in the qualitative review of computational science literature. It becomes apparent that this research area relies on the definition of taxonomies that would benefit from clarification through collaboration with social science research. For instance, the inclusion of stereotypes against women as part of the types of misogyny raises the question of whether the concept of misogyny should be reserved for the most extreme forms of hatred or should encompass the wide range of sexist attitudes and gender symbolic constructions derived from a patriarchal culture.

The main conclusion drawn from this work is that research across different disciplines is addressing a new facet of misogyny, a revitalised version of outdated beliefs about women’s inferiority that circulate in novel forms within the online realm. Understanding the characteristics and functions of this new expression of misogyny poses a challenge that necessitates an interdisciplinary approach, leveraging the strengths of different areas of knowledge to effectively address it.

The above-mentioned lack of collaboration between different areas prevents the establishment of connections that would enrich the analysis of the way misogyny is disseminated today in both the virtual and real world. For example, social science knowledge in combination with computational discourse analysis or NLP technologies could be used to study the connections and similarities between agents disseminating misogyny online and mainstream social actors such as political parties or religious organisations. In the same way, the similarity between misogynist discourses and those of left-leaning feminists in open battle against other fractions of the feminist movement could also be monitored and would allow for a more complex view of the phenomenon. For both approaches, it is necessary that social science knowledge strongly rooted in the study of social relations be combined with the new methodologies that computer science offers for the analysis of discourse produced naturally in digital or real communicative exchanges, such as in parliaments, rallies or interviews.

Data availability

Data sharing is not applicable to this research, as no data were generated. The analysed data were retrieved from the commercial Web of Science (WOS) and Scopus databases, following the search procedure detailed in the Supplementary Material .

https://www.theguardian.com/technology/2015/jan/11/gamergate-a-brief-history-of-a-computer-age-war , https://time.com/3510381/gamergate-faq/

https://independenttechresearch.org/letter-twitters-new-api-plans-will-devastate-public-interest-research/

https://independenttechresearch.org/reddit-survey-results/

Alghamdi R, Alfalqi K (2015) A survey of topic modeling in text mining. Int J Adv Comput Sci Appl 6(1):147–153

Google Scholar  

Allen A (2021) Feminist perspectives on power. In: Zalta E (ed) The Stanford encyclopedia of philosophy. Metaphysics Research Lab, Stanford University

Anzovino M, Fersini E, Rosso P (2018) Automatic identification and classification of misogynistic language on Twitter. In: Silberztein M, Atigui F, Kornyshova E et al (eds) Natural language processing and information systems (NLDB) 2018. Lecture notes in computer science, vol 10859. Springer, Cham, pp. 57–64

Aria M, Cuccurullo C (2017) bibliometrix: an R-tool for comprehensive science mapping analysis. J Informetr 11(4):959–975

Article   Google Scholar  

Attanasio G, Pastor E (2020) PoliTeam @ AMI: improving sentence embedding similarity with misogyny lexicons for automatic misogyny identification in Italian tweets. In: Proceedings of the 7th evaluation campaign of natural language processing and speech tools for Italian (EVALITA 2020) (eds Basile V, Croce D, Maro M, Passaro LC) CEUR workshop proceedings, vol 2765, Accademia University Press, Torino

Banet-Weiser S, Miltner KM (2016) #MasculinitySoFragile: culture, structure, and networked misogyny. Fem Media Stud 16(1):171–174

Bassignana E, Basile V, Patti V (2018) Hurtlex: a multilingual lexicon of words to hurt. In: Proceedings of the fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Torino, December 10–12, 2018, (eds Cabrio E, Mazzei A, Tamburini F) CEUR Workshop Proceedings, vol 2253, Accademia University Press, Torino, Italy

Bates L (2021) Men who hate women: from incels to pickup artists: the truth about extreme misogyny and how it affects us all. Sourcebooks, Naperville, IL

Blei DM (2012) Probabilistic topic models. Commun ACM 55(4):77–84

Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3(1):993–1022

Bosch NV, Gil-Juarez A (2021) Un acercamiento situado a las violencias machistas online y a las formas de contrarrestarlas. Rev Estud Fem 29(3). https://doi.org/10.1590/1806-9584-2021V29N374588

Byerly CM(2020) Incels online reframing sexual violence Commun Rev 23(4):290–308

Carian EK(2022) "We’re all in this together”: leveraging a personal action frame in two men’s rights forums Mobilization 27(1):47–68

Chang W(2022) The monstrous-feminine in the incel imagination: Investigating the representation of women as “femoids” on /r/Braincels Fem Media Stud 22(2):254–270

Article   MathSciNet   Google Scholar  

Chen GM, Pain P, Chen VY(2020) ‘You really have to have a thick skin’: a cross-cultural perspective on how online harassment influences female journalists Journalism 21(7):877–895

Chiril P, Pamungkas E, Benamara F (2022) Emotionally informed hate speech detection: a multi-target perspective. Cogn Comput 14(1):322–352

Citron DK (2014) Hate crimes in cyberspace. Harvard University, Cambridge, MA

Book   Google Scholar  

Code L (2000) Encyclopedia of feminist theories. Routledge, London

Davis S, Santillana M(2019) From the streets to the screen to nowhere: Las morras and the fragility of networked digital activism Westminst Pap Commun Cult 14(1):18–32

Dickel V, Evolvi G (2022) “Victims of feminism”: exploring networked misogyny and #MeToo in the manosphere. Fem Media Stud https://doi.org/10.1080/14680777.2022.2029925

Döring N, Mohseni MR(2019) Male dominance and sexism on YouTube: results of three content analyses Fem Media Stud 19(4):512–524

Dynel M(2020) Vigilante disparaging humour at r/IncelTears: humour as critique of incel ideology Language Commun 74:1–14

Faith B(2022) Tackling online gender-based violence; understanding gender, development, and the power relations of digital spaces Gend Technol Dev 26(3):325–340

Farci M, Righetti N(2019) Italian men’s rights activism and online backlash against feminism Rass Ital Sociol 60(4):765–781

Fersini E, Nozza D, Rosso P (2018) Overview of the EVALITa 2018 task on automatic misogyny identification (AMI). In: CEUR workshop proceedings (eds Caselli T, Novielli N, Patti V, Rosso P) vol 2263 (CEUR-WS), Accademia University Press, Torino

Fersini E, Gasparini F, Rizzi G et al (2022) SemEval-2022 Task 5: multimedia automatic misogyny identification. In: Proceedings of the 16th international workshop on Semantic Evaluation (SemEval-2022) (eds Emerson G, Schluter N, Stanovsky G. et al) Association for Computational Linguistics, pp. 533–549

Francine Tougas F, Brown R, Beaton AM (1999) Neosexism among women: the role of personally experienced social mobility attempts. Personal Soc Psychol Bull 25(12):1487–1497

Fuchs T, Schäfer F(2021) Normalizing misogyny: hate speech and verbal abuse of female politicians on Japanese Twitter Jpn Forum 33(4):553–579

Garcìa-Mingo E, Fernàndez SD, Tomàs-Forte S (2022) (Re)configurando el imaginario sobre la violencia sexual desde el antifeminismo: el trabajo ideològico de la manosfera española. Polit Sociedad 59(1). https://doi.org/10.5209/poso.80369

Ghaffari S(2022) Discourses of celebrities on Instagram: digital femininity, self-representation and hate speech Crit Discourse Stud 19(2):161–178

Gill R (2016) Post-postfeminism?: new feminist visibilities in postfeminist times. Fem Media Stud 16(4):610–630

Ging D, Siapera E (2018) Special issue on online misogyny. Fem Media Stud 18(4):515–524

Glick P, Fiske ST (1997) Hostile and benevolent sexism: measuring ambivalent sexist attitudes toward women. Psychol Women Q 21(1):119–135

Greenacre M, Blasius J (2006) Multiple correspondence analysis and related methods. Chapman and Hall/CRC, New York

Guest E, Vidgen B, Mittos A et al An expert annotated dataset for the detection of online misogyny. In: Proceedings of the 16th conference of the European Chapter of the Association for Computational Linguistics: main volume. Association for Computational Linguistics, pp. 1336–1350 (2021)

Guhl J, Davey J (2020) A safe space to hate: white supremacist mobilisation on telegram. Technical report. ISD Global

Halpin M(2022) Weaponized subordination: how incels discredit themselves to degrade women Gend Soc 36(6):813–837

Hopton K, Langer S(2022) “Kick the XX out of your life”: an analysis of the manosphere’s discursive constructions of gender on Twitter Fem Psychol 32(1):3–22

Hu Y, Boyd-Graber J, Satinoff B (2014) Interactive topic modeling. Mach Learn 95(3):423–469

Jane EA(2014) Back to the kitchen, cunt: speaking the unspeakable about online misogyny Continuum 28(4):558–570

Jane EA (2015) Flaming? What flaming? The pitfalls and potentials of researching online hostility. Ethics Inf Technol 17(1):65–87

Jane EA(2016) Online misogyny and feminist digilantism Continuum 30(3):284–297

Jane EA (2017) Misogyny online: a short (and Brutish) history. Sage, London

Jeffreys S (2005) Beauty and misogyny: harmful cultural practices in the West. Routledge, New York, NY, USA

Jenson J, De Castell S(2021) Patriarchy in play: video games as gendered media ecologies Explor Media Ecol 20(2):195–212

Jeong E, Lee J(2018) We take the red pill, we confront the DickTrix: online feminist activism and the augmentation of gendered realities in South Korea Fem Media Stud 18(4):705–717

Jones C, Trott V, Wright S(2020) Sluts and soyboys: MGTOW and the production of misogynistic online harassment New Media Soc 22(10):1903–1921

Jukes A (1993) Why men hate women. Free Association Books Ltd, London

Kim J(2017) #iamafeminist as the “mother tag”: feminist identification and activism against misogyny on Twitter in South Korea Fem Media Stud 17(5):804–820

Kim Y(2021) Mirroring misogyny in Hell Choson: Megalia, womad, and Korea’s feminism in the age of digital populism Eur J Korean Stud 20(2):101–133

Koirala S(2020) Female journalists’ experience of online harassment: a case study of Nepal Media Commun 8(1):47–56

Koller V, Heritage F(2020) Incels, in-groups, and ideologies. The representation of gendered social actors in a sexuality-based online community J Language Sex 9(2):152–178

Kurasawa F, Rondinelli E, Kilicaslan G(2021) Evidentiary activism in the digital age: on the rise of feminist struggles against gender-based online violence Inf Commun Soc 24(14):2174–2194

Kwarteng J, Perfumi S, Farrell T et al (2022) Misogynoir: challenges in detecting intersectional hate. Soc Netw Anal Min 12(1). https://doi.org/10.1007/s13278-022-00993-7

Lindsay A(2022) Swallowing the Black Pill: Involuntary Celibates’ (Incels) anti-feminism within Digital Society Int J Crime Justice Soc Democr 11(1):210–224

Lonsway KA, Fitzgerald LF (1994) Rape myths: in review. Psychol Women Q 18:133–164

Lonsway KA, Fitzgerald LF (1995) Attitudinal antecedents of rape myth acceptance: a theoretical and empirical reexamination. J Personal Soc Psychol 68(4):704–711

Manne K (2017) Down girl: the logic of misogyny. Oxford Academic, New York

Mantilla K (2013) Gendertrolling: misogyny adapts to new media. Fem Stud 39(2):563–571

Marwick AE, Caplan R(2018) Drinking male tears: language, the manosphere, and networked harassment Fem Media Stud 18(4):543–559

Massanari A (2017) #Gamergate and The Fappening: how Reddit’s algorithm, governance, and culture support toxic technocultures. New Media Soc 19(3):329–346

Massanari AL (2020) Gamergate. John Wiley & Sons, Ltd, pp. 1–5

MATLAB (2022) MATLAB version R2022b. The MathWorks Inc., Natick, MA

Menzie L(2022) Stacys, Beckys, and Chads: the construction of femininity and hegemonic masculinity within incel rhetoric Psychol Sex 13(1):69–85

Millet K (1970) Sexual Politics. Doubleday, New York, NY

Moloney ME, Love TP (2018) Assessing online misogyny: perspectives from sociology and feminist media studies. Sociol Compass 12(5):e12,577

Moon YE, Kim HH, Park D (2022) “Can I become a true feminist?”: an interpretive analysis on the mirroring experience of young Korean women. Fem Media Stud https://doi.org/10.1080/14680777.2022.2042830

Mulki H, Ghanem B (2022) Working notes of the workshop Arabic Misogyny Identification (ArMI-2021). In: Proceedings of the 13th annual meeting of the Forum for Information Retrieval Evaluation. Association for Computing Machinery, New York, NY, USA, pp. 7–8

O’Donnell C, Shor E(2022) "This is a political movement, friend”: why “incels” support violence Br J Sociol 73(2):336–351

Article   PubMed   Google Scholar  

Ostini J, Hopkins S (2015) Online harassment is a form of violence. The Conversation 8: 1–4

Pamungkas EW, Cignarella AT, Basile V et al (2018) Automatic identification of misogyny in English and Italian Tweets at EVALITA 2018 with a multilingual hate lexicon. In: Proceedings of the sixth evaluation campaign of natural language processing and speech tools for Italian. Final workshop (EVALITA 2018), Turin co-located with the fifth Italian conference on Computational Linguistics (CLiC-it 2018), Turin (eds Caselli T, Novielli N, Patti V, Rosso P) December 12–13, 2018, CEUR Workshop Proceedings, vol 2263, Accademia University Press, Torino, Italy

Phipps EB, Montgomery F(2022) "Only YOU Can Prevent This Nightmare, America”: Nancy Pelosi as the monstrous-feminine in Donald Trump’s YouTube attacks Women’s Stud Commun 45(3):316–337

Price DJ (1963) Little science, big science. Columbia University Press, New York

Rego R(2018) Changing forms and platforms of misogyny: sexual harassment of women journalists on twitter Media Watch 9(3):472–485

Ritchie J(2013) Creating a monster: Online media constructions of Hillary Clinton during the Democratic Primary Campaign, 2007-8 Fem Media Stud 13(1):102–119

Rodrìguez-Sanchez FJ, Carrillo-de Albornoz J, Plaza L (2021) Overview of EXIST 2021: sEXism Identification in Social neTworks. Proces Leng Nat 67:195–207

Rubio Martìn MJ, Gordo Lòpez A (2021) La perspectiva tecnosocial feminista como antídoto para la misoginia online. Rev Esp Sociol 30(3). https://doi.org/10.22325/fes/res.2021.64

Saluja N, Thilaka DN(2021) Women leaders and digital communication: gender stereotyping of female politicians on Twitter J Content Community Commun 13(7):227–241

Scotto di Carlo G(2023) An analysis of self-other representations in the incelosphere: between online misogyny and self-contempt Discourse Soc 34(1):3–21

Semenzin S, Bainotti L (2020) The use of telegram for non-consensual dissemination of intimate images: gendered affordances and the construction of masculinities. Soc Media + Soc 6(4):2056305120984,453

Shesterina AM, Fedosova OA (2021) Promotion of feminist ideas in instagram texts. Vestnik Mosk Univ Seriya 10 Zhurnalistika 2021(4):114–134

Shushkevich E, Cardiff J (2019) Automatic misogyny detection in social media: a survey. Comput Sist 23(4):1159–1164

Sills S, Pickens C, Beach K(2016) Rape culture and social media: young critics and a feminist counterpublic Fem Media Stud 16(6):935–951

Silva-Paredes D, Ibarra Herrera D (2022) Resisting anti-democratic values with misogynistic abuse against a Chilean right-wing politician on Twitter: the #CamilaPeluche incident Discourse Commun 16(4):426–444

Speckhard A, Ellenberg M, Morton J, Ash A (2021) Involuntary Celibates’ experiences of and grievance over sexual exclusion and the potential threat of violence among those active in an Online Incel Forum. J Strateg Secur 14(2):89–121

Steyvers M, Griffiths T (2006) Probabilistic topic models. In: Landauer T, McNamara D, Dennis S et al (eds) Latent semantic analysis: a road to meaning. Lawrence Erlbaum

Sultana S, Sarker J, Bosu A (2021) A Rubric to identify misogynistic and sexist texts from software developer communications. In: Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). Association for Computing Machinery, New York, NY, USA

Suominen A, Toivanen H (2016) Map of science with topic modeling: comparison of unsupervised learning and human-assigned subject classification. J Assoc Inf Sci Technol 67(10). https://doi.org/10.1002/asi.23596

Tomkinson S, Harper T(2015) The position of women in video game culture: Perez and Day’s Twitter Incident Continuum 29(4):617–634

Tontodimamma A, Nissi E, Sarra A (2021) Thirty years of research into hate speech: topics of interest and their evolution. Scientometrics 126(157–179):69–81

Tranchese A, Sugiura L(2021) "I don’t hate all women, just those stuck-up bitches”: how incels and mainstream pornography speak the same extreme language of misogyny Violence Against Women 27(14):2709–2734

Article   PubMed   PubMed Central   Google Scholar  

van Dijk TA (1998) Editorial: discourse and ideology. Discourse Soc 9(3):307–308

VandenBos GR (ed) (2015) APA dictionary of psychology, 2nd edn. American Psychological Association

Wagner A(2022) Tolerating the trolls? Gendered perceptions of online harassment of politicians in Canada Fem Media Stud 22(1):32–47

Whiley LA, Walasek L, Juanchich M(2023) Contributions to reducing online gender harassment: social re-norming and appealing to empathy as tried-and-failed techniques Fem Psychol 33(1):83–104

Wright S, Trott V, Jones C(2020) ‘The pussy ain’t worth it, bro’: assessing the discourse and structure of MGTOW Inf Commun Soc 23(6):908–925

Yang S, Lee K(2022) The intertextuality and interdiscursivity of “mirroring” in South Korean cyberfeminist posts Discourse Soc 33(5):671–689

Zeinert P, Inie N, Derczynski L (2021) Annotating online misogyny. In: Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (vol 1: Long Papers). Association for Computational Linguistics, pp. 3181–3197

Download references

Acknowledgements

This research was undertaken as part of the ICOMIC (Identifying and Counteracting Online Misogyny in Cyberspace) Project funded by EU Next Generation, MUR-Fondo Promozione e Sviluppo-DM 737/2021

Author information

Authors and affiliations.

Department of Legal and Social Sciences, G. d’Annunzio University of Chieti-Pescara, Chieti-Pescara, Italy

Lara Fontanella & Alice Tontodimamma

PRHLT, Universitat Politècnica de València, Valencia, Spain

Berta Chulvi

Social Psychology Department, Universitat de València, Valencia, Spain

Department of Neuroscience, Imaging and Clinical Sciences, G. d’Annunzio University of Chieti-Pescara, Chieti-Pescara, Italy

Elisa Ignazzi

Department of Philosophical, Pedagogical and Economic-Quantitative Sciences, G. d’Annunzio University of Chieti-Pescara, Chieti-Pescara, Italy

Annalina Sarra

You can also search for this author in PubMed   Google Scholar

Contributions

LF and BC contributed to the study conception and design, with LF leading the study supervision. LF, BC, and AS contributed to the writing of the manuscript. LF, EI, and AT developed the dataset and conducted the statistical analysis. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Lara Fontanella .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethics approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

Informed consent was not required as the study did not involve human participants.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary material, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Fontanella, L., Chulvi, B., Ignazzi, E. et al. How do we study misogyny in the digital age? A systematic literature review using a computational linguistic approach. Humanit Soc Sci Commun 11 , 478 (2024). https://doi.org/10.1057/s41599-024-02978-7

Download citation

Received : 14 July 2023

Accepted : 18 March 2024

Published : 02 April 2024

DOI : https://doi.org/10.1057/s41599-024-02978-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Dermatol
• v.59(2); Mar-Apr 2014

Understanding and Evaluating Systematic Reviews and Meta-analyses

Michael bigby.

From the Department of Dermatology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA 02215, USA

A systematic review is a summary of existing evidence that answers a specific clinical question, contains a thorough, unbiased search of the relevant literature, explicit criteria for assessing studies and structured presentation of the results. A systematic review that incorporates quantitative pooling of similar studies to produce an overall summary of treatment effects is a meta-analysis. A systematic review should have clear, focused clinical objectives containing four elements expressed through the acronym PICO (Patient, group of patients, or problem, an Intervention, a Comparison intervention and specific Outcomes). Explicit and thorough search of the literature is a pre-requisite of any good systematic review. Reviews should have pre-defined explicit criteria for what studies would be included and the analysis should include only those studies that fit the inclusion criteria. The quality (risk of bias) of the primary studies should be critically appraised. Particularly the role of publication and language bias should be acknowledged and addressed by the review, whenever possible. Structured reporting of the results with quantitative pooling of the data must be attempted, whenever appropriate. The review should include interpretation of the data, including implications for clinical practice and further research. Overall, the current quality of reporting of systematic reviews remains highly variable.

Introduction

A systematic review is a summary of existing evidence that answers a specific clinical question, contains a thorough, unbiased search of the relevant literature, explicit criteria for assessing studies and structured presentation of the results. A systematic review can be distinguished from a narrative review because it will have explicitly stated objectives (the focused clinical question), materials (the relevant medical literature) and methods (the way in which studies are assessed and summarized).[ 1 , 2 ] A systematic review that incorporates quantitative pooling of similar studies to produce an overall summary of treatment effects is a meta-analysis.[ 1 , 2 ] Meta-analysis may allow recognition of important treatment effects by combining the results of small trials that individually might lack the power to consistently demonstrate differences among treatments.[ 1 ]

With over 200 speciality dermatology journals being published, the amount of data published just in the dermatologic literature exceeds our ability to read it.[ 3 ] Therefore, keeping up with the literature by reading journals is an impossible task. Systematic reviews provide a solution to handle information overload for practicing physicians.

Criteria for reporting systematic reviews have been developed by a consensus panel first published as Quality of Reporting of Meta-analyses (QUOROM) and later refined as Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).[ 4 , 5 ] This detailed, 27-item checklist contains items that should be included and reported in high quality systematic reviews and meta-analyses. The methods for understanding and appraising systematic reviews and meta-analyses presented in this paper are a subset of the PRISMA criteria.

The items that are the essential features of a systematic review include having clear objectives, explicit criteria for study selection, an assessment of the quality of included studies, criteria for which studies can be combined, appropriate analysis and presentation of results and practical conclusions that are based on the evidence evaluated [ Table 1 ]. Meta-analysis is only appropriate if the included studies are conceptually similar. Meta-analyses should only be conducted after a systematic review.[ 1 , 6 ]

Criteria for evaluating a systematic review or the meta-analysis

A Systematic Review Should Have Clear, Focused Clinical Objectives

A focused clinical question for a systematic review should contain the same four elements used to formulate well-built clinical questions for individual studies, namely a Patient, group of patients, or problem, an Intervention, a Comparison intervention and specific Outcomes.[ 7 ] These features can be remembered by the acronym PICO. The interventions and comparison interventions should be adequately described so that what was done can be reproduced in future studies and in practice. For diseases with established effective treatments, comparisons of new treatments or regimens to established treatments provide the most useful information. The outcomes reported should be those that are most relevant to physicians and patients.[ 1 ]

Explicit and Thorough Search of the Literature

A key question to ask of a systematic review is: “Is it unlikely that important, relevant studies were missed?” A sound systematic review can be performed only if most or all of the available data are examined. An explicit and thorough search of the literature should be performed. It should include searching several electronic bibliographic databases including the Cochrane Controlled Trials Registry, which is part of the Cochrane Library, Medline, Embase and Literatura Latino Americana em Ciências da Saúde. Bibliographies of retrieved studies, review articles and textbooks should be examined for studies fitting inclusion criteria. There should be no language restrictions. Additional sources of data include scrutiny of citation lists in retrieved articles, hand-searching for conference reports, prospective trial registers (e.g., clinical trials.gov for the USA and clinical trialsregister.eu for the European union) and contacting key researchers, authors and drug companies.[ 1 , 8 ]

Reviews should have Pre-defined Explicit Criteria for what Studies would be Included and the Analysis should Include Only those Studies that Fit the Inclusion Criteria

The overwhelming majority of systematic reviews involve therapy. Randomized, controlled clinical trials should therefore be used for systematic reviews of therapy if they are available, because they are generally less susceptible to selection and information bias in comparison with other study designs.[ 1 , 9 ]

Systematic reviews of diagnostic studies and harmful effects of interventions are increasingly being performed and published. Ideally, diagnostic studies included in systematic reviews should be cohort studies of representative populations. The studies should include a criterion (gold) standard test used to establish a diagnosis that is applied uniformly and blinded to the results of the test(s) being studied.[ 1 , 9 ]

Randomized controlled trials can be included in systematic reviews of studies of adverse effects of interventions if the events are common. For rare adverse effects, case-control studies, post-marketing surveillance studies and case reports are more appropriate.[ 1 , 9 ]

The Quality (Risk of Bias) of the Primary Studies should be Critically Appraised

The risk of bias of included therapeutic trials is assessed using the criteria that are used to evaluate individual randomized controlled clinical trials. The quality criteria commonly used include concealed, random allocation; groups similar in terms of known prognostic factors; equal treatment of groups; blinding of patients, researchers and analyzers of the data to treatment allocation and accounting for all patients entered into the trial when analyzing the results (intention-to-treat design).[ 1 ] Absence of these items has been demonstrated to increase the risk of bias of systematic reviews and to exaggerate the treatment effects in individual studies.[ 10 ]

Structured Reporting of the Results with Quantitative Pooling of the Data, if Appropriate

Systematic reviews that contain studies that have results that are similar in magnitude and direction provide results that are most likely to be true and useful. It may be impossible to draw firm conclusions from systematic reviews in which studies have results of widely different magnitude and direction.[ 1 , 9 ]

Meta-analysis should only be performed to synthesize results from different trials if the trials have conceptual homogeneity.[ 1 , 6 , 9 ] The trials must involve similar patient populations, have used similar treatments and have measured results in a similar fashion at a similar point in time.

Once conceptual homogeneity is established and the decision to combine results is made, there are two main statistical methods by which results are combined: random-effects models (e.g., DerSimonian and Laird) and fixed-effects models (e.g., Peto or Mantel-Haenszel).[ 11 ] Random-effects models assume that the results of the different studies may come from different populations with varying responses to treatment. Fixed-effects models assume that each trial represents a random sample of a single population with a single response to treatment [ Figure 1 ]. In general, random-effects models are more conservative (i.e., random-effects models are less likely to show statistically significant results than fixed-effects models). When the combined studies have statistical homogeneity (i.e., when the studies are reasonably similar in direction, magnitude and variability), random-effects and fixed-effects models give similar results.

Fixed-effects models (a) assume that each trial represents a random sample (colored curves) of a single population with a single response to treatment. Random-effects models (b) assume that the different trials’ results (colored curves) may come from different populations with varying responses to treatment.

The point estimates and confidence intervals of the individual trials and the synthesis of all trials in meta-analysis are typically displayed graphically in a forest plot [ Figure 2 ].[ 12 ] Results are most commonly expressed as the odds ratio (OR) of the treatment effect (i.e., the odds of achieving a good outcome in the treated group divided by the odds of achieving a good result in the control group) but can be expressed as risk differences (i.e., difference in response rate) or relative risk (probability of achieving a good outcome in the treated group divided by the probability in the control group). An OR of 1 (null) indicates no difference between treatment and control and is usually represented by a vertical line passing through 1 on the x-axis. An OR of greater or less than 1 implies that the treatment is superior or inferior to the control respectively.

Annotated results of a meta-analysis of six studies, using random effects models reported as odd ratios using MIX version 1.7 (Bax L, Yu LM, Ikeda N, Tsuruta H, Moons KGM. Development and validation of MIX: comprehensive free software for meta-analysis of causal research data. BMC Med Res Methodol http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1626481/ ). The central graph is a typical Forest Plot

The point estimate of individual trials is indicated by a square whose size is proportional to the size of the trial (i.e., number of patients analyzed). The precision of the trial is represented by the 95% confidence interval that appears in Forest Plots as the brackets surrounding point estimate. If the 95% confidence interval (brackets) does not cross null (OR of 1), then the individual trial is statistically significant at the P = 0.05 level.[ 12 ] The summary value for all trials is shown graphically as a parallelogram whose size is proportional to the total number of patients analyzed from all trials. The lateral tips of the parallelogram represent the 95% confidence interval and if they do not cross null (OR of 1), then the summary value of the meta-analysis is statistically significant at the P = 0.05 level. ORs can be converted to risk differences and numbers needed to treat (NNTs) if the event rate in the control group is known [ Table 2 ].[ 13 , 14 ]

Deriving numbers needed to treat from a treatment's odds ratio and the observed or expected event rates of untreated groups or individuals

The difference in response rate and its reciprocal, the NNT, are the most easily understood measures of the magnitude of the treatment effect.[ 1 , 9 ] The NNT represents the number of patients one would need to treat in order to achieve one additional cure. Whereas the interpretation of NNT might be straightforward within one trial, interpretation of NNT requires some caution within a systematic review, as this statistic is highly sensitive to baseline event rates.[ 1 ]

For example, if a treatment A is 30% more effective than treatment B for clearing psoriasis and 50% of people on treatment B are cleared with therapy, then 65% will clear with treatment A. These results correspond to a rate difference of 15% (65-50) and an NNT of 7 (1/0.15). This difference sounds quite worthwhile clinically. However if the baseline clearance rate for treatment B in another trial or setting is only 30%, the rate difference will be only 9% and the NNT now becomes 11 and if the baseline clearance rate is 10%, then the NNT for treatment A will be 33, which is perhaps less worthwhile.[ 1 ]

Therefore, NNT summary measures within a systematic review should be interpreted with caution because “control” or baseline event rates usually differ considerably between studies.[ 1 , 15 ] Instead, a range of NNTs for a range of plausible control event rates that occur in different clinical settings should be given, along with their 95% confidence intervals.[ 1 , 16 ]

The data used in a meta-analysis can be tested for statistical heterogeneity. Methods to tests for statistical heterogeneity include the χ 2 and I.[ 2 , 11 , 17 ] Tests for statistical heterogeneity are typically of low power and hence detecting statistical homogeneity does not mean clinical homogeneity. When there is evidence of heterogeneity, reasons for heterogeneity between studies – such as different disease subgroups, intervention dosage, or study quality – should be sought.[ 11 , 17 ] Detecting the source of heterogeneity generally requires sub-group analysis, which is only possible when data from many or large trials are available.[ 1 , 9 ]

In some systematic reviews in which a large number of trials have been performed, it is possible to evaluate whether certain subgroups (e.g. children versus adults) are more likely to benefit than others. Subgroup analysis is rarely possible in dermatology, because few trials are available. Subgroup analyses should always be pre-specified in a systematic review protocol in order to avoid spurious post hoc claims.[ 1 , 9 ]

The Importance of Publication Bias

Publication bias is the tendency that studies that show positive effects are more likely to be published and are easier to find.[ 1 , 18 ] It results from allowing factors other than the quality of the study to influence its acceptability for publication. Factors such as the sample size, the direction and statistical significance of findings, or the investigators’ perception of whether the findings are “interesting,” are related to the likelihood of publication.[ 1 , 19 , 20 ] Negative studies with small sample size are less likely to be published.[ 1 , 19 , 20 ] Studies published are often dominated by the pharmaceutical company sponsored trials of new, expensive treatments often compared with the placebo.

For many diseases, the studies published are dominated by drug company-sponsored trials of new, expensive treatments. Such studies are almost always “positive.”[ 1 , 21 , 22 ] This bias in publication can result in data-driven systematic reviews that draw more attention to those medicines. Systematic reviews that have been sponsored directly or indirectly by industry are also prone to bias through over-inclusion of unpublished “positive” studies that are kept “on file” by that company and by not including or not finishing registered trials whose results are negative.[ 1 , 23 ] The creation of study registers (e.g. http://clinicaltrials.gov ) and advance publication of research designs have been proposed as ways to prevent publication bias.[ 1 , 24 , 25 ] Many dermatology journals now require all their published trials to have been registered beforehand, but this policy is not well policed.[ 1 ]

Language bias is the tendency for studies that are “positive” to be published in an English-language journal and be more quickly found than inconclusive or negative studies.[ 1 , 26 ] A thorough systematic review should therefore not restrict itself to journals published in English.[ 1 ]

Publication bias can be detected by using a simple graphic test (funnel plot), by calculating the fail-safe N, Begg's rank correlation method, Egger regression method and others.[ 1 , 9 , 11 , 27 , 28 ] These techniques are of limited value when less than 10 randomized controlled trials are included. Testing for publication bias is often not possible in systematic reviews of skin diseases, due to the limited number and sizes of trials.[ 1 , 9 ]

Question-driven systematic reviews answer the clinical questions of most concern to practitioners. In many cases, studies that are of most relevance to doctors and patients have not been done in the field of dermatology, due to inadequate sources of independent funding.[ 1 , 9 ]

The Quality of Reporting of Systematic Reviews

The quality of reporting of systematic reviews is highly variable.[ 1 ] One cross-sectional study of 300 systematic reviews published in Medline showed that over 90% were reported in specialty journals. Funding sources were not reported in 40% of reviews. Only two-thirds reported the range of years that the literature was searched for trials. Around a third of reviews failed to provide a quality assessment of the included studies and only half of the reviews included the term “systematic review” or “meta-analysis” in the title.[ 1 , 29 ]

The Review should Include Interpretation of the Data, Including Implications for Clinical Practice and Further Research

The conclusions in the discussion section of a systematic review should closely reflect the data that have been presented within that review. Clinical recommendations can be made when conclusive evidence is found, analyzed and presented. The authors should make it clear which of the treatment recommendations are based on the review data and which reflect their own judgments.[ 1 , 9 ]

Many reviews in dermatology, however, find little evidence to address the questions posed. The review may still be of value even if it lacks conclusive evidence, especially if the question addressed is an important one.[ 1 , 30 ] For example, the systematic review may provide the authors with the opportunity to call for primary research in an area and to make recommendations on study design and outcomes that might help future researchers.[ 1 , 31 ]

Source of Support: Nil

Conflict of Interest: Nil.

• Share full article

Advertisement

Supported by

Daniel Kahneman, Who Plumbed the Psychology of Economics, Dies at 90

He helped pioneer a branch of the field that exposed hard-wired mental biases in people’s economic behavior. The work led to a Nobel.

By Robert D. Hershey Jr.

Daniel Kahneman, who never took an economics course but who pioneered a psychologically based branch of that field that led to a Nobel in economic science in 2002, died on Wednesday. He was 90.

His death was confirmed by his partner, Barbara Tversky. She declined to say where he died.

Professor Kahneman, who was long associated with Princeton University and lived in Manhattan, employed his training as a psychologist to advance what came to be called behavioral economics. The work, done largely in the 1970s, led to a rethinking of issues as far-flung as medical malpractice, international political negotiations and the evaluation of baseball talent, all of which he analyzed, mostly in collaboration with Amos Tversky , a Stanford cognitive psychologist who did groundbreaking work on human judgment and decision-making. (Ms. Tversky, also a professor of psychology at Stanford , had been married to Professor Tversky, who died in 1996. She and Professor Kahneman became partners several years ago.)

As opposed to traditional economics, which assumes that human beings generally act in fully rational ways and that any exceptions tend to disappear as the stakes are raised, the behavioral school is based on exposing hard-wired mental biases that can warp judgment, often with counterintuitive results.

“His central message could not be more important,” the Harvard psychologist and author Steven Pinker told The Guardian in 2014, “namely, that human reason left to its own devices is apt to engage in a number of fallacies and systematic errors, so if we want to make better decisions in our personal lives and as a society, we ought to be aware of these biases and seek workarounds. That’s a powerful and important discovery.”

Professor Kahneman delighted in pointing out and explaining what he called universal brain “kinks.” The most important of these, the behaviorists hold, is loss-aversion: Why, for example, does the loss of $100 hurt about twice as much as the gaining of $100 brings pleasure?

Among its myriad implications, loss-aversion theory suggests that it is foolish to check one’s stock portfolio frequently, since the predominance of pain experienced in the stock market will most likely lead to excessive and possibly self-defeating caution.

Loss-aversion also explains why golfers have been found to putt better when going for par on a given hole than for a stroke-gaining birdie. They try harder on a par putt because they dearly want to avoid a bogey, or a loss of a stroke.

Mild-mannered and self-effacing, Professor Kahneman not only welcomed debate on his ideas; he also enlisted the help of adversaries as well as colleagues to perfect them. When asked who should be considered the “father” of behavioral economics, Professor Kahneman pointed to the University of Chicago economist Richard H. Thaler , a younger scholar (by 11 years) whom he described in his Nobel autobiography as his second most important professional friend, after Professor Tversky.

“I’m the grandfather of behavioral economics,” Professor Kahneman allowed in a 2016 interview for this obituary, in a restaurant near his home in Lower Manhattan.

This new school of thought did not get its first major public airing until 1985, in a conference at the University of Chicago Graduate School of Business, a bastion of traditional economics.

Professor Kahneman’s public reputation rested heavily on his 2011 book “Thinking, Fast and Slow,” which appeared on best-seller lists in science and business. One commentator, the essayist, mathematical statistician and former option trader Nassim Nicholas Taleb, author of the influential book on improbability “The Black Swan,” placed “Thinking” in the same league as Adam Smith’s “The Wealth of Nations” and Sigmund Freud’s “The Interpretation of Dreams.”

The author Jim Holt, writing in The New York Times Book Review , called “Thinking” “an astonishingly rich book: lucid, profound, full of intellectual surprises and self-help value.”

Shane Frederick, a professor at the Yale School of Management and a Kahneman protégé, said by email in 2016 that Professor Kahneman had “helped transform economics into a true behavioral science rather than a mere mathematical exercise in outlining the logical entailments of a set of often wildly untenable assumptions.”

An Accessible Writer

Professor Kahneman propagated his findings with an appealing writing style, using illustrative vignettes with which even lay readers could engage.

Professor Kahneman wrote, for example, that Professor Thaler had inspired him to study, as an experiment, the so-called mental accounting of someone who arrives at the theater and realizes that he has lost either his ticket or the cash equivalent. Professor Kahneman found that people who lost the cash would still buy a ticket by some means, while those who lost an already purchased ticket would more likely go home.

Professor Thaler won the 2017 Nobel in economic science — officially the Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel. Professor Kahneman shared his 2002 Nobel with Vernon L. Smith of George Mason University in Virginia. “Had Tversky lived, he would certainly have shared the Nobel with Kahneman, his longtime collaborator and dear friend,” Professor Holt wrote in his 2011 Times review . Professor Tversky died in 1996 at 59.

Much of Professor Kahneman’s work is grounded in the notion — which he did not originate but organized and advanced — that the mind operates in two modes: fast and intuitive (mental activities that we’re more or less born with, called System One), or slow and analytical, a more complex mode involving experience and requiring effort (System Two).

Others have personified these mental modes as Econs (rational, analytical people) and Humans (emotional, impulsive and prone to exhibit unconscious mental biases and an unwise reliance on dubious rules of thumb). Professor Kahneman and Professor Tversky used the word “heuristics” to describe these rules of thumb. One is the “halo effect,” where in observing a positive attribute of another person one perceives other strengths that aren’t really there.

“Before Kahneman and Tversky, people who thought about social problems and human behavior tended to assume that we are mostly rational agents,” the Times columnist David Brooks wrote in 2011 . “They assumed that people have control over the most important parts of their own thinking. They assumed that people are basically sensible utility-maximizers, and that when they depart from reason it’s because some passion like fear or love has distorted their judgment.”

But Professors Kahneman and Tversky, he went on, “yielded a different vision of human nature.”

As Mr. Brooks described it: “We are players in a game we don’t understand. Most of our own thinking is below awareness.” He added: “Our biases frequently cause us to want the wrong things. Our perceptions and memories are slippery, especially about our own mental states. Our free will is bounded. We have much less control over ourselves than we thought.”

The work of Professor Kahneman and Professor Tversky, he concluded, “will be remembered hundreds of years from now.”

In the Shadow of Nazis

Daniel Kahneman was born on March 5, 1934, into a family of Lithuanian Jews who had emigrated to France to the early 1920s. After France fell to Nazi Germany in World War II, Daniel, like other Jews, was forced to wear a Star of David on the outside of his clothing. His father, the research chief in a chemical factory, was seized and interned at a way station before deportation to an extermination camp, but he was then released under mysterious circumstances. The family escaped to the Riviera and then to central France, where they lived in a converted chicken coop.

Daniel’s father died just before D-Day, in June 1944, and Daniel, by then an eighth-grader, and his sister, Ruth, wound up in British-controlled Palestine with their mother, Rachel. (Daniel had been born in Tel Aviv during an extended visit with relatives by his mother.)

He graduated from the Hebrew University of Jerusalem with a major in psychology, completing his college studies in two years. In 1954, after the founding of the state of Israel, he was drafted into the Israeli Defense Forces as a second lieutenant.

After a year as a platoon leader, he was transferred to the psychology branch, where he was given occasional assignments to assess candidates for officer training.

The unit’s ability to predict performance, however, was so poor that he coined the term “illusion of validity,” meaning a cognitive bias in which one displays overconfidence in the accuracy of one’s judgments. Two decades later this “illusion” became one of the most frequently cited elements in psychology literature.

He married Irah Kahan in Israel, and they soon set off for the University of California, Berkeley, where he had been granted a fellowship. He earned his Ph.D. in psychology there. He returned to Israel to teach at Hebrew University from 1961 to 1977. The marriage ended in divorce. (Professor Kahneman held dual citizenships, in the United States and Israel.)

In 1978, Professor Kahneman married Anne Treisman, a noted British psychologist who shared his interest in the study of attention, which was the chief subject of his early work. The two of them ran a lab and wrote papers together. In 2013 she received the National Medal of Science from President Barack Obama. She died in 2018. He and Ms. Treisman had long been friends with the Tverskys.

In addition to Ms. Tversky, he is survived by a son and daughter from his first marriage, Michael Kahneman and Lenore Shoham; two stepdaughters from his second marriage, Jessica and Deborah Treisman; two stepsons from the same marriage, Daniel and Stephen Treisman; three grandchildren; and four step-granddaughters. He lived in Greenwich Village for many years.

It was in Jerusalem, while developing a training course for Air Force flight instructors, that Professor Kahneman had “the most satisfying Eureka experience of my career,” as he wrote in an autobiographical sketch for the Nobel committee.

He had started to preach the traditional view that to promote learning, praise is more effective than punishment. But a seasoned colleague insisted otherwise, telling him, as Professor Kahneman recalled:

“On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.”

The colleague had insisted — and convinced Professor Kahneman — that statistically people may do very well in something in one instance or very poorly in another, but that in the end they tend to regress to the mean, or average.

“This was a joyous moment, in which I learned an important truth about the world,” Professor Kahneman wrote. “Because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them.”

His collaboration with Professor Tversky — their peak productive years were 1971 to 1981 — was exceptionally close, so much so that it inspired the author Michael Lewis to write a book about them, “The Undoing Project : A Friendship That Changed Our Minds” (2016).

“Amos and I shared the wonder of together owning a goose that could lay golden eggs — a joint mind that was better than our separate minds,” Professor Kahneman wrote in his Nobel autobiography. Later, in “Thinking,” he wrote, “The pleasure we found in working together made us exceptionally patient; it is much easier to strive for perfection when you are never bored.”

Mr. Lewis reported that the two men worked on a single typewriter, often amid uproarious laughter and shouts in Hebrew and English, and that they had sometimes flipped a coin to determine whose name would be listed first on a paper.

But they also feuded, particularly when Professor Kahneman thought he was being denied proper credit. One falling-out lasted years, ending finally with a reconciliation. Professor Kahneman was solicitous during his colleague’s final illness (he died of metastatic melanoma) and was his main eulogist at his funeral in 1996.

One product of their collaboration was a finding that overconfidence in conjunction with optimism is an extremely common bias, which leads people to think that wars are quickly winnable and that building projects will be completed on budget. But Professor Kahneman and Professor Tversky considered such bias necessary in the end for capitalism to function.

Professor Kahneman’s North American career included teaching posts at the University of British Columbia and Berkeley before he joined the Princeton University faculty in 1993.

His most recent book is “Noise: A Flaw in Human Judgment” (2021), written with Cass Sunstein and Olivier Sibony. In The Times Book Review, Steven Brill called it a “tour de force of scholarship and clear writing.”

The book looks at how human judgment often varies wildly even among specialists, as reflected in judicial decisions, insurance premiums, medical diagnoses and corporate decisions, as well as in many other aspects of life.

And it distinguishes between predictable biases — a judge, for example, who consistently sentences Black defendants more harshly — and what the authors call “noise”: less explainable decisions resulting from what they define as “unwanted variability in judgments.” In one example, the authors report that doctors are more likely to order cancer screenings for patients they see early in the morning than late in the afternoon.

The book, like his others, was an outgrowth of Professor Kahneman’s lifelong quest to understand how the human mind works — what thought processes lead people to make the kinds of decisions and judgments they do as they navigate a complex world. And toward the end of his life he acknowledged that so much more was to be known.

In an interview with Kara Swisher on her Times podcast “Sway” in 2021, he said, “If I were starting my career now, I would be choosing between artificial intelligence and neuroscience, because those are now particularly exciting ways of looking at human nature.”

Robert D. Hershey Jr. , a longtime reporter who wrote about finance and economics for The Times, died in January. Alex Traub contributed reporting.