
Threats to Validity
 Construct
 External
 Internal
 Statistical Conclusion

Threats to Statistical Conclusion Validity
 Low Statistical Power
 Violated Assumptions of Statistical Tests
 Fishing and the Error Rate Problem
 Unreliability of Measures
 Restriction of Range
 Unreliability of Treatment Implementation
 Extraneous Variance in the Experimental Setting
 Heterogeneity of Units
 Inaccurate Effect Size Estimation

SCV: Low Statistical Power
An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.

SCV: Violated assumptions of Statistical Tests
Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance of an effect.

SCV: Fishing and the Error Rate Problem
Repeated tests for significant relationships, if uncorrected for the number of tests, can artifactually inflate statistical significance

SCV: Unreliability of Measures
Measurement error weakens the relationship between two variables and strengthens or weakens the relationship among three or more variables.

SCV: Restriction of Range
Reduced range on a variable usually weakens the relationship between it and another variable.

SCV: Unreliability of Treatment Implementation
If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.

SCV: Extraneous Variance in the Experimental Setting
Some features of an experimental setting may inflate error, making detection of an effect more difficult.

SCV: Heterogeneity of Units
Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult.

SCV: Inaccurate Effect Size Estimation
Some statistics systematically overestimate or under estimate the size of an effect.

Ways to increase statistical power
 use matching, stratifying, or blocking
 measure, and correct for covariates
 use larger sample sizes
 use equal cell sample sizes
 improve measurement (e.g. increase range of measurements/reduce dichotomized variables, add additional waves of measurement)
 increase the strength of treatment (e.g. increase dose differential)
 Use a withinparticipants design
 Use homogeneous participants selected to be responsive to treatment
 Reduce random setting irrelevancies
 Ensure that powerful statistical tests are used and their assumptions are met. (e.g. transforming the data)

Internal Validity Definition
Inferences about whether observed variation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured.

Threats to Construct Validity
 Inadequate Explication of Constructs
 Construct Confounding
 MonoOperation Bias
 MonoMethod Bias
 Confounding Constructs with Levels of Constructs
 Treatment Sensitive Factorial Structure
 Reactive SelfReport Changes
 Reactivity to the Experimental Situation
 Experimenter Expediencies
 Novelty and Disruption Effects
 Compensatory Equalization
 Compensatory Rivalry
 Resentful Demoralization
 Treatment Diffusion

