quals: clinical data mining part IV

• estimate parameters
• Y = f(X1, ..., Xn, noise, parameters)
• aims to infer properties of a population given a smaller, observable sample
• could this function have generated the output I see?

yes-no using goodness-of-fit tests and residual examination

• significance: is there a measurable difference?
• effect size: is the measured difference relevant?

• If many samples collected and the 95% confidence interval computed, 95% of the intervals would contain the true value of the parameter, theta
• 95% confidence interval does NOT mean that there is a 95% probability that the interval contains the true value of the parameter theta

providing a best guess of a parameter as a function over data points form some distribution

• Method to estimate the intercept and slope of a line quantifying the relationship between at least one independent (predictor) variable and one dependent (outcome) variable
• Y=α+βX+random error

• linear model with more than one predictor
• allows adjustment for possible confounders
• Interpret each coefficient as the change in the outcome per unit increase in that variable, with all other variables held constant

fits a model for a categorical outcome variable

• Comparing response rates in cancer patients among treatment arms adjusted for age and duration of disease
• Comparing the proportion of hypertensive patients between two treatment arms adjusted for age, cholesterol level and exercise habits

Multivariate regression for a time-to-event outcome

• Relates the time that passes before some event occurs to one or more covariates that may be associated with that event
• Assumes that the effects of the predictor variables upon survival are constant over time and are multiplicative

• Increase the bias of linear regression a little to reduce the variance by a lot
• ridge, lasso, elastic net

• Model space = piecewise constant functions with disjoint rectangular regions
• robust to outliers, fast to fit, will ignore useless features
• not good predictors because high variance, decision boundary is sensitive to what the training data looks like

• bagging: fit many trees to bootstrap-resampled versions of the training data, use majority vote as prediction
• random forest: at each split, only consider a random subset of feature to split
• boosting: fit trees to successively reweighted versions of the data, accounting for errors made by previous trees, use weighted majority vote as prediction
• boosting and random forests are among the best ML methods
• boosting is slightly better than random forest, but has more parameters to tune

• falsely rejecting the null hypothesis when there is no effect
• happens 1 in 20 times when p-value threshold is 0.05

failing to reject the null hypothesis when there actually is an effect

• inverse of type II error
• chance of detecting a true effect if it exists

• define a null hypothesis
• determine the null distribution for the variable to be observed
• conduct experiment to measure variable under conditions of interest
• summarize the variable as a point estimate of the unobserved parameter (e.g. mean)
• determine the probability that the observation was by chance given the null distribution
• if the probability is low (less than 5%), reject the null hypothesis, declare a measureable difference from the null

the rate of experiencing the outcome

the chance of experiencing the outcome relative to not experiencing the outcome

• measure of the risk of an outcome in a group compared to another group
• Not possible to calculate because prevalence of outcome in the samp

• A measure of the odds of the outcome in a group compared to a control group
• Can be calculated from case-control study data
• Output of logistic regression (you get the odds ratio for free)

• learn a function
• f(x) - that operates on x to predict y
• what is the function that could have generated the output I see?

measured by predictive accuracy

• accuracy: how often do i get it right?
• actionability: i got it right, so what?

• how likely are you ot have the disease if you test positive?
• TP / TP + FP

• How likely are you to not have the disease if you test negative?
• TN / TN + FN

• How likely are you to test positive if you have the disease?
• TP / TP + FN

• How likely are you to test negative if you do not have the disease?
• TN / TN + FP

• If you want to do causal analysis on observational data, then you need to find two people that are the same besides the fact that one person got the exposure and the other did not. 
• In other words, we want to find two people that had the same probability of receiving treatment, but only one of them actually did receive treatment

• find sub-groups: clustering
• find themes: look for patterns, decomposition

Factors that can cause or prevent the outcome of interest, at not intermediate variables, and are associated with the factors under investigation

• consider confounding factors
• replication: over time, across sites, using different study designs
• quantify the "instability" of analysis: i.e. variance in the face of alternative study design and data source choices
• test for, and quantify non-stationarity in the data
• examine multiple dimensions of performance: calibration, AUPRC, effect size, estimated FDR, etc.