-
Threats to Validity
- Construct
- External
- Internal
- Statistical Conclusion
-
Threats to Statistical Conclusion Validity
- Low Statistical Power
- Violated Assumptions of Statistical Tests
- Fishing and the Error Rate Problem
- Unreliability of Measures
- Restriction of Range
- Unreliability of Treatment Implementation
- Extraneous Variance in the Experimental Setting
- Heterogeneity of Units
- Inaccurate Effect Size Estimation
-
SCV: Low Statistical Power
An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.
-
SCV: Violated assumptions of Statistical Tests
Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance of an effect.
-
SCV: Fishing and the Error Rate Problem
Repeated tests for significant relationships, if uncorrected for the number of tests, can artifactually inflate statistical significance
-
SCV: Unreliability of Measures
Measurement error weakens the relationship between two variables and strengthens or weakens the relationship among three or more variables.
-
SCV: Restriction of Range
Reduced range on a variable usually weakens the relationship between it and another variable.
-
SCV: Unreliability of Treatment Implementation
If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.
-
SCV: Extraneous Variance in the Experimental Setting
Some features of an experimental setting may inflate error, making detection of an effect more difficult.
-
SCV: Heterogeneity of Units
Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult.
-
SCV: Inaccurate Effect Size Estimation
Some statistics systematically overestimate or under estimate the size of an effect.
-
Ways to increase statistical power
- use matching, stratifying, or blocking
- measure, and correct for covariates
- use larger sample sizes
- use equal cell sample sizes
- improve measurement (e.g. increase range of measurements/reduce dichotomized variables, add additional waves of measurement)
- increase the strength of treatment (e.g. increase dose differential)
- Use a within-participants design
- Use homogeneous participants selected to be responsive to treatment
- Reduce random setting irrelevancies
- Ensure that powerful statistical tests are used and their assumptions are met. (e.g. transforming the data)
-
Internal Validity Definition
Inferences about whether observed variation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured.
-
Threats to Construct Validity
- Inadequate Explication of Constructs
- Construct Confounding
- Mono-Operation Bias
- Mono-Method Bias
- Confounding Constructs with Levels of Constructs
- Treatment Sensitive Factorial Structure
- Reactive Self-Report Changes
- Reactivity to the Experimental Situation
- Experimenter Expediencies
- Novelty and Disruption Effects
- Compensatory Equalization
- Compensatory Rivalry
- Resentful Demoralization
- Treatment Diffusion
|
|
