How do I perform statistical analysis and interpret the p-value in a clinical study?

How to Perform Statistical Analysis and Interpret P-Values

Statistical analysis should be planned before data collection begins, with clear specification of the primary outcome, significance level (typically α=0.05), and appropriate statistical tests based on your data type and study design. 1

Planning Your Statistical Analysis

Before Starting Your Study

• Define your research question clearly - this determines everything else in your statistical approach 1
• Specify your primary endpoint before collecting data, as sample size calculations depend on this 1
• Determine your significance threshold (typically p<0.05 for two-sided tests) and document whether you'll use one-sided or two-sided testing 1
• Calculate required sample size based on expected effect size, desired power (typically 80-90%), and significance level 1
• Plan for missing data by specifying handling methods in advance 1
• Decide on multiple comparison corrections if you have multiple secondary outcomes 1

Choosing the Right Statistical Test

Your choice of statistical test depends on three critical factors: your data type (continuous vs. categorical), data distribution (normal vs. non-normal), and whether observations are independent or paired. 2, 3

For Continuous Data:

• Normally distributed, independent samples: Use two-sample t-test 2
• Normally distributed, paired/dependent samples: Use paired t-test 2
• Non-normally distributed, independent samples: Use Mann-Whitney U test 2
• Non-normally distributed, paired samples: Use Wilcoxon signed rank test 2

For Categorical Data:

• Independent groups with expected values ≥5: Use Chi-square test 2
• Independent groups with expected values <5: Use Fisher's exact test 2
• Paired binary data: Use McNemar test 2

For Meta-Analyses:

• Use random effects models (DerSimonian-Laird method) when substantial heterogeneity (I² >50%) is expected 1
• Weight studies by inverse-variance to account for sample size differences 1
• Report heterogeneity using I² statistic, where 25%, 50%, and 75% indicate low, medium, and high heterogeneity respectively 1

Understanding and Reporting P-Values

What P-Values Mean

A p-value represents the probability of obtaining results at least as extreme as those observed if the null hypothesis (no difference) were true. 1, 4

• P<0.05 means less than 5% probability the observed difference occurred by chance alone 4
• P between 0.05-0.01 represents modest evidence against the null hypothesis 4
• P<0.001 represents strong evidence against the null hypothesis 4

Critical Reporting Requirements

Never report p-values alone - always report effect sizes (odds ratios, hazard ratios, mean differences) with 95% confidence intervals alongside p-values. 1

• Report precise p-values to two decimal places when p>0.01, three decimal places when p<0.01, or as "p<0.001" for very small values 1, 4
• Report two-sided p-values unless your study design explicitly assumes one-sided testing 1
• Include confidence intervals because they show both the direction and magnitude of effect, unlike p-values which only indicate statistical significance 1, 4

Common Pitfalls to Avoid

• Don't use "trend" to describe p-values close to 0.05 - results are either statistically significant or not based on your pre-specified threshold 1
• Don't include p-values in baseline characteristics tables for randomized trials - any differences are due to chance by design 1
• Do include p-values for observational studies comparing baseline characteristics between groups 1
• Avoid p-values for secondary/subgroup analyses where proper type I error controls aren't in place - use point estimates and confidence intervals instead 1

Reporting Your Results

In the Methods Section

Describe statistical methods with enough detail that someone with access to your data could reproduce your results. 1

• State your study objectives and patient population clearly 1
• Specify your analysis software and version (e.g., STATA version 18, R version 4.1.1) as different programs may produce slightly different results 1
• Document your significance level (typically 0.05) 1
• Describe handling of missing data 1

In the Results Section

Present continuous variables as mean ± standard deviation for normally distributed data, or median with interquartile range for skewed distributions. 1

• Report categorical outcomes as frequencies and percentages with one decimal place when denominator >200 1
• Clearly state denominators used for percentage calculations 1
• Present effect estimates (risk ratios, odds ratios, hazard ratios) with 95% confidence intervals 1, 4
• Use forest plots for subgroup analyses, including point estimates, confidence intervals, and sample sizes 1

Understanding Clinical vs. Statistical Significance

Statistical significance (p<0.05) does not automatically mean clinical importance - even tiny differences become statistically significant with large sample sizes. 1

• Define minimal clinically important difference before starting your study 1
• Evaluate both statistical significance and clinical relevance when interpreting results 4
• Consider that with large samples, even clinically meaningless differences can achieve p<0.05 1