Does Heterogeneity Decrease Quality Rating in JBI Critical Appraisal?
No, heterogeneity itself does not directly decrease the quality rating when using the JBI Critical Appraisal Checklist for Systematic Reviews—the JBI checklist assesses methodological quality and risk of bias in the review process, not the statistical heterogeneity of included studies.
Understanding the Distinction
The JBI Critical Appraisal Checklist for Systematic Reviews evaluates the methodological rigor of how a systematic review was conducted, not the characteristics of the data being synthesized 1. The checklist contains 11 questions assessing specific domains to determine potential risk of bias, with responses of "yes," "no," "unclear," or "not applicable" 1. Quality ratings are calculated by dividing the number of "yes" responses by total scored items, with cutoffs of ≥70% for low risk, 50-69% for moderate risk, and <50% for high risk of bias 1.
Where Heterogeneity Actually Matters
Heterogeneity is addressed separately in the GRADE assessment of certainty of evidence, not in the JBI critical appraisal of review quality 2. The GRADE approach evaluates five domains: overall study risk of bias, inconsistency (which includes heterogeneity), indirectness, imprecision, and publication bias 2.
Key distinction:
- JBI checklist = assesses whether reviewers conducted their systematic review properly
- GRADE/certainty assessment = evaluates the strength and consistency of the evidence itself
How Heterogeneity Should Be Handled
When substantial statistical heterogeneity is present (I² >50%, Chi² P<0.1, or Tau² >0), best practice dictates 2:
- Meta-analysis should not be performed
- Results should be reported separately
- Random effects models should be used if pooling is attempted with moderate heterogeneity
- Fixed effect models only when minimal heterogeneity exists
The presence of heterogeneity and how reviewers respond to it can affect JBI quality ratings indirectly—if reviewers fail to appropriately assess or address heterogeneity in their methods, this reflects poor methodological quality 3, 4.
Common Pitfalls
Avoid conflating these concepts:
- High heterogeneity in study results ≠ poor quality systematic review
- A well-conducted review can identify and appropriately handle heterogeneous data
- Conversely, a poorly conducted review might inappropriately pool heterogeneous studies, which would lower the JBI quality rating because it reflects methodological failure 5
The I² statistic quantifies variation across studies beyond chance, with values <25% considered low 3, 4. However, reviewers should exercise caution when few studies are available, as I² has large uncertainty in these situations 3, 4.
Bottom line: Heterogeneity affects the certainty of evidence and appropriateness of meta-analysis, but the JBI checklist evaluates whether the systematic review authors properly identified, reported, and handled that heterogeneity—not the heterogeneity itself 2, 1.