A biostatistical approach describes how we can realistically and feasibly execute a research idea to produce meaningful data, and then translate those data into interpretations and decisions. The way data analysis is conducted, and the results are presented can be a sensitive topic for reviewers. So, during Medical Research Evaluations in a journal, editors carefully check the statistical analysis and result section.
Based on the review and thematic framework, we summarize the medical study statistical analysis steps reported in the statistical analysis section.
Step 1: Specify Study Objective Type and Outcomes (Overall Approach):
The study objective type provides the role of important variables for a specified outcome in statistical analyses and the overall approach of the model building and model reporting steps in a study.
According to the statistical framework, problems can be classified as descriptive or inferential/analytical/confirmatory. For example, analytical and prognostic problems in epidemiology are broadly classified as association, explanation and prediction. In medical research, there are six categories of study objectives: (1) exploratory, (2) association, (3) causal, (4) intervention, (5) prediction, and (6) clinical decision models.
An objective exploratory study is a type of medical research determinant study commonly known as a risk factor or correlate study. An exploratory study considers all covariates equally important for the outcome. An exploratory study aims to present the results of a model that gives higher accuracy after satisfying all assumptions. In association studies, the investigator defines predefined exposures of interest for the outcome; variables other than exposures are also considered covariates. Associative studies present the adjusted association between exposure and outcome. An objective causal study uses the conceptual framework to determine the impact of exposures on outcomes. Each variable in this study objective should have a predefined role (exposures, confounders, mediators, covariates, and predictors). Medical research with a causal objective is called an explanatory or confirmatory study. After assessing the model’s fitness in the conceptual framework, the direct or indirect effects of exposures are presented.
In medical research, interventional studies are often randomized or non-randomized clinical trials that assess the effectiveness of interventions. In the intervention objective model, all variables except the intervention are treated as a nuisance. The goal is to show the direct effect of the intervention on the outcomes. Predictive studies aim to identify the best variables that can predict outcomes. Clinical decision models are prognostic models that use high-dimensional data at various levels for risk stratification, classification, and prediction. All variables are input features in this model. The goal of training, testing, and validation data sets aim to present a high-accuracy decision tool. Before proceeding with statistical analyses, biostatisticians or applied researchers should discuss the study’s objective type. A conceptual model framework should also be prepared regardless of the study objective.
Examples:
A study aimed to develop a predictive model for predicting Alzheimer’s disease progression. However, in the medical study statistical analysis of this study did not internally or externally validate the model’s performance as per the requirement of an objective predictive study. In another study, investigators were interested in determining an association between metabolic syndrome and the hepatitis C virus. However, the authors did not specify the outcome of the analysis and produced conflicting associations with different analyses. Thus, the outcome should be specified as per the study objective type.
Step 2: Specify Effect Size Measure According to Study Design (Interpretation and Practical Value):
The study design describes the selection of study participants and the data collecting method as a function of either exposure or outcome. The research study design largely impacts the proper application of effect size measurement, tabular presentation of results, and significant level. In cohort or clinical trial research designs, individuals are recruited according to their exposure status and their development of the outcome is monitored.
These research designs may provide numerous outcomes, incidence or incidence density, and risk ratio (RR) or hazard models are favored for analysis. In case-control research, participant selection is contingent on outcome status. Therefore, this research can only have one outcome, and odds ratio (OR) models are chosen for analysis.
In a cross-sectional research design, neither outcomes nor exposures are restricted. All data are gathered concurrently and may be evaluated using the theoretically equivalent prevalence ratio approach. The reporting of impact size measurements also depends on the research’s objective type. Unlike other objective kinds, predictive models often offer regression coefficients or the weight of variables in the model rather than association measurements. In medical study statistical analysis, there are both correlations and discordances between OR and RR measurements. Due to its consistency and symmetry OR is preferred by several researchers in investigations involving frequent occurrences.
Step 3: Specify the Study Hypothesis, Report P-values, and Estimate the Interval (Interpretation and Decision):
The clinical hypothesis assesses the formal claims provided in the study’s objectives. In contrast, the statistical hypothesis gives information on the characteristics and statistics of the population used to assess formal claims. Typically the p-value and 95% confidence interval (CI) is used to assess inferences about the research hypothesis. A lower p-value suggests that the findings are consistent with the null hypothesis. However, since the p-value is a conditional probability, it can never indicate whether the null hypothesis should be accepted or rejected. To increase the validity of the results, some different p-value calculation methodologies have been presented. Adaptation of these alternative methodologies is necessary solely for explanation-focused research. Although it is suggested that precise p-values be published in research papers, p-values give no information on effect size.
In contrast to p-values, the confidence interval (CI) offers a confidence interval of the effect size that covers the true effect size if the research is repeated and may be used to evaluate if the findings are statistically significant. The p-value and the 95% confidence interval offer to complement information and should thus be included in the medical study statistical analysis section. Frequently, researchers examine many comparisons or hypotheses. Consequently, the direction and significance level may be altered so that the findings may be considered statistically significant. In addition, studies may have several main outcomes, necessitating a multiplicity-adjusted significance level. Therefore, in their main analysis, all research should include an interval estimate of the effect size/regression coefficient.
Step 4: Account for DGP in the Statistical Analysis (Accuracy):
As part of the study design, it is required to identify the techniques for participant selection and outcome measurement in the various design settings. This particular design element was designated as the DGP. Understanding DGP is important for determining an appropriate model for analyzing outcome distributions and establishing the model’s premises and units of analysis. DGP encompasses data production and measurement processes, such as the number of measurements after random, complicated, sequential, pragmatic and systematic selection. Moreover, DGP needs a sampling, clustering, pragmatic or systematic review context (participants are selected from published studies).
DGP also relies on the assessed outcomes in an unpaired, paired or mixed scenario. This study provides information on the outcomes of exposure generation processes employing quantitative or categorical variables, quantitative values obtained from labs or validated instruments, and self-reported or administered tests resulting in various data distributions, such as individual, mixed-type, mixed, or mixed latent distributions. Depending on the DGP employed, study data may include incomplete/partial measurements, time-varying measurements, surrogate measures, latent measures, imbalances, unidentified confounders, instrument variables, correlated responses, clustering levels, qualitative or mixed data outcomes, competing events, individual and higher-level variables, etc.
DGP is required for a medical study statistical analysis, determination of standard errors and subsequent computation of p-values, the generalizability of results and graphical representation of data. To adequately account for DGP in the analyses, researchers and biostatisticians must communicate about all aspects of participant selection and data collection including measures, measurement occasions, and study devices.
Examples :
Middle-aged men participated in a randomized controlled trial to examine the effects of fresh fruit and komatsuna juice on metabolic markers. This research has been criticized for for using erroneous statistical procedures inconsistent with the study’s overall design.
In addition, another research revealed that 80% of published studies using the Korean National Health and Nutrition Examination Survey did not include a survey sample framework in their statistical analyses, resulting in biased estimates and improper conclusions. Another research highlighted the need to maintain methodological standards while evaluating National Inpatient Sample data. Over 50% of papers in the top 25% of physiology journals did not mention whether paired or unpaired t-tests were used in medical study statistical analysis, indicating a lack of transparency in statistical analysis reporting. In another investigation, data indicating mistakes that are not in accordance with DGP were also identified. DGP separates the delay between the development of symptoms and the commencement of therapy into three categories: patient/primary delay, secondary delay, and tertiary delay. Additionally, a count data model would be suitable for evaluating the number of malignant lymph nodes. In some research, the data were not examined according to DGP.
Step 5: Apply EBB Methods Specific to Study Design Features and DGP (Efficiency and Robustness):
Due to the continual expansion in the development of strong statistical methods, several methods for analyzing different kinds of data have been developed. Consequently, several strategies may be used for a particular problem. Nevertheless, the performance of each approach differs, resulting in diverse methods among applied researchers.
Variable practices may also be caused by a lack of agreement on statistical approaches in the literature, ignorance and the absence of established statistical rules. Nevertheless, it may be challenging to distinguish if a particular approach was used owing to its robustness, a lack of understanding, the availability of statistical tools to apply a suitable alternative method, the aim to achieve predicted results, or ignorance of model diagnostics. In order to prevent diverse practices, the selection of statistical techniques and the reporting of its findings at each level of data analysis must adhere to EBB norms. It is challenging for applied researchers to find the best suitable statistical methods at each phase. Therefore, we urge researchers to include biostatisticians early in fundamental, clinical, population, translational and database research.
Moreover, we urge biostatisticians to provide guidelines, checklists, and teaching resources to promote the EBB idea. We designed the medical study statistical analysis and methods in biomedical research (SAMBR) recommendations for applied researchers to employ EBB data analysis methodologies. For the EBB practice to provide accurate and impartial outcomes, it is essential to use cutting-edge, rigorous methodology. The effectiveness of a statistical approach is governed by its assumptions and DGP. Consequently, researchers may seek to define the selection of certain models in the main analysis under the EBB.
Examples:
Even though SAMBR checklists provide details of evidence-based preferred methods for each study design or objective, we have presented a simplified version of evidence-based preferred methods for common statistical analyses. Several examples of inefficient methods in the literature need to be in accordance with EBB practice.
Step 6: Report Variable Selection Method in the Multivariable Analysis According to the Study Objective Type (Unbiased):
There are several applications of multivariate analysis including association, prediction or classification, risk stratification, adjustment, propensity score development and effect size estimation. Various biological, clinical, behavioral, and environmental factors may directly influence the relationship between exposure and outcome. In order to interpret findings accurately and unbiasedly, most health studies require multivariate analyses. It is recommended that analysts develop an adjusted model if the sample size permits. There is a misconception that the analysis of RCTs can be done without adjustment. Prognostic variables may need to be adjusted when analyzing RCTs. The first step in building a model is the entry of variables once the appropriate parametric or non-parametric regression model has been determined. Due to the lack of preference for exposures, a backward automated approach can be used for variable selection in the exploratory model-building process after any significant variables have been included at 25% in the unadjusted model. A fully adjusted association model should include a manual selection of covariates based on the relevance of the variables. The selection and adjustment of variables should be guided by clinically guided methods in a causal model. In a non-randomized interventional study, propensity score methods should be utilized to eliminate confounding effects. It is possible to adjust any prognostic variable in the final propensity score-adjusted multivariable model.
In a randomized study, however, the prognostic variables should be adjusted. Maintaining the event per variable (EVR) is important to avoid overfitting in any modeling. Therefore, some association and explanatory studies may require screening of variables, which can be achieved using a stepwise backward method that must be clarified in the medical study statistical analysis. Ideally, a model that utilizes the variables producing the highest level of accuracy should be used in a predictive study. Various methods may be used to identify the optimal variables including random forest, bootstrapping and machine learning. Differences in results may result from different variable selection and adjustment methods. The statistical analysis section should describe the screening process of variables and their adjustments in the final multivariable model.
Examples:
A study evaluating the effectiveness of hydroxychloroquine (HDQ) found that patients who received HDQ experienced unfavorable outcomes (intubation or death) compared to those who did not receive HDQ (hazard ratio (HR): 2.37, 95%CI 1.84 to 3.02) in an unadjusted analysis. However, as appropriate with the interventional objective model, the propensity score-adjusted analyses found no significant association between HDQ use and unfavorable events (HR: 1.04, 95%CI 0.82 to 1.32); multivariable and other propensity score-adjusted analyses further confirmed this. In observational studies, results should only be interpreted based on multivariable analyses if this is feasible. According to a recent study, approximately 6% of multivariable analyses based on logistic regression or Cox regression used an inappropriate selection method of variables. This practice was often observed in studies that a biostatistician expert did not conduct. A review of 316 articles published in high-impact Chinese medical journals revealed that 30.7% did not report the selection of variables in multivariable models. If the objective study type had been classified, this inappropriate practice might have been detected more frequently. Even though an adjusted analysis may produce a more efficient estimate than an unadjusted analysis, it is uncommon to report an adjusted analysis in RCTs. According to a study assessing the effect of the preemptive intervention on developmental outcomes, the intervention significantly reduced the severity of the symptoms associated with an autism spectrum disorder. However, Ware criticized it because it failed to report non-significant results in its unadjusted analyses. It is recommended that unadjusted estimates be included in all studies, especially randomized controlled trials.
Step 7: Provide Evidence for Exploring Effect Modifiers (Applicability):
Variables that modify the effect of exposure on the outcome are called effect modifiers or modifiers of interacting variables.
Exploring the effect modifiers in multivariable analyses helps in the following:
- Determining the applicability/generalizability of findings in the overall or specific subpopulation,
- Generating new hypotheses,
- Providing explanations for uninterpretable findings between unadjusted and adjusted analyses,
- Guiding as to whether we should present combined or separate models for each specific subpopulation, and
- Explaining heterogeneity in treatment effect.
In many cases, investigators present adjusted stratified results based on the presence or absence of an effect modifier.
The stratified findings (subgroups) according to each effect modifier may be presented if the exposure interacts with multiple variables statistically or conceptually. Due to the smaller sample sizes within each stratum, stratified analyses significantly reduce the power of the study. There may be significant results due to the inflating of type I errors. Therefore, it may be more appropriate to present a multivariate analysis using an interaction term rather than a stratified analysis with an effect modifier. A quantitative variable may be a potential modifier of an exposure-outcome relationship in some cases. It is not recommended to categorize quantitative variables in such a situation unless there is no clinically meaningful threshold. The categorization of quantitative variables should only be conducted with a clinically meaningful cut-off or a hypothesis that can be supported. The results of an objective exploratory study may reveal any possible interaction; however, it is important to consider the clinical implications when interpreting the results. Similarly, some objective models may have more than one exposure or intervention. Whenever there is an interaction effect, the association between each exposure and the level of the other exposure should be presented via adjusted analyses.
Example:
A review of 428 articles from MEDLINE on the quality of reporting using statistical analyses of three commonly used regression models (linear, logistic, and Cox) found that only 18.5% of the published articles provided interaction analyses, even though interaction analyses can provide valuable information.
Step 8: Evaluate Hypotheses Focusing on the Outcome Distribution, Linearity, Multicollinearity, Sparsity, and Overfitting (Reliability):
During medical study statistical analysis, the evaluation and reporting of model diagnostics are crucial for determining the model’s effectiveness, validity, and utility. Model diagnostics ensure that assumptions related to the model are met and evaluate sparsity, linearity, outcome distribution, multicollinearity, and overfitting. In addition, model-specific assumptions are necessary. These include distribution fit in other types of continuous and count models, normal residuals, heteroscedasticity, independence of errors in linear regression, proportionality odds assumption in ordinal logistic regression, and proportionality in Cox regression. Before choosing a suitable model, it is also important to consider sparsity. Numerous zero observations in the data collection are a sign of sparsity. It is challenging to interpret the effect size when there is sparsity.
Most parametric and semiparametric models, except machine learning models, demand a linear relationship between independent variables and a functional output form. Therefore, linearity should be evaluated in all model objectives using a multivariable polynomial. Similarly, selecting the right result distribution is necessary for model creation in all objective study models.
All objective models can benefit from the assessment of multicollinearity. In multivariable analysis, the evaluation of EVR can be used to avoid the overfitting issue.
Step 9: Report Type of Primary and Sensitivity Analyses (Consistency):
Several considerations and assumptions must be evaluated, assessed, and validated throughout the research process. Some initial assumptions, procedures, and errors may not be correctable. Nonetheless, extra information collected throughout the research and data processing, including data distribution provided at the end of the study, may permit new considerations that must be verified in the medical study statistical analysis. In research investigations, a sensitivity analysis may be used to evaluate the consistency of research results by adjusting the outcome or exposure definition, the study population, accounting for missing data, model assumptions, variables, and their forms and accounting for protocol compliance. Statistical studies must define the goal and supporting analyses to distinguish the primary results from the supporting findings. Unlike secondary, intermediate, or subgroup analyses, sensitivity analyses are unique. Sometimes, secondary analyses refer to data analysis for secondary outcomes. In contrast, data analyses of current research are known as interim analyses, while data analyses depending on patient characteristics are referred to as subgroup analyses.
Step 10: Provide Methods for Summarizing, Displaying and Interpreting Data (Transparency and Usability):
Data presentation covers data summary, data display, and data obtained from statistical model analyses. The purpose of data summaries is to comprehend the distribution of outcome status and other characteristics in the sample by exposure status and outcome status. Therefore, it is suggested that all research designs report data column-wise according to exposure status. In addition, all research designs, except for case-control studies, should show outcomes in rows.
A summary statistic should offer as much information as possible on the data distribution following the DGP and variable type. The primary purpose of regression analysis or statistical model results is to explain its interpretations and consequences. Therefore, results should be presented following the study’s objectives. For example, in the objective determinant model, it may be advantageous to provide both the unadjusted and adjusted relationships of each component with the result.
The explanatory objective model may favor unadjusted and adjusted primary exposure effects on the outcome. A prognostic model may provide the user with the final prediction models to use these models to predict an outcome. A final multivariable model must be described as an exploratory objective model with R2 or area under the curve (AUC). It is crucial to evaluate associational and interventional models’ internal validity using various sensitivity and validation techniques.
An objective causal model must be finalized with improved fit indices (in terms of R2 or AUC, Akaike information criterion, Bayesian information criterion, fit index, and root mean square error).
The objective predictive type must report the model’s performance regarding R2 or AUC in both training and validation data sets. There are several applications for data visualization. For example, bar diagrams, histograms, frequency polygons, and box plots may illustrate data distribution. In addition to cluster bar diagrams, scatter dot plots, stacked bar diagrams, and Kaplan-Meier plots may be used to illustrate various comparisons.
A scatter plot or scatter matrix may be used to illustrate correlation or model evaluation. Heatmaps and line plots may be used to illustrate grouping and patterns. With fitted models, margin plots may highlight the influence of predictors. Using a forest plot for the comparison of regression model effect sizes. Although the purpose of data presentation is to emphasize important concerns or discoveries in the research, it must comply with DGP and variable types and be user-friendly. The effect size measure, the design of the study, and the presented hypotheses are crucial to interpreting the results. Sometimes, variables need standardization. In the medical study statistical analysis phase of research projects, appropriate data reporting and interpretation procedures should be outlined per the study design, hypothesis, and effect size measure.
