HPEB 818

  1. Threats to Validity
    • Construct
    • External
    • Internal
    • Statistical Conclusion
  2. Threats to Statistical Conclusion Validity
    • Low Statistical Power
    • Violated Assumptions of Statistical Tests
    • Fishing and the Error Rate Problem
    • Unreliability of Measures
    • Restriction of Range
    • Unreliability of Treatment Implementation
    • Extraneous Variance in the Experimental Setting
    • Heterogeneity of Units
    • Inaccurate Effect Size Estimation
  3. SCV: Low Statistical Power
    An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.
  4. SCV: Violated assumptions of Statistical Tests
    Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance of an effect.
  5. SCV: Fishing and the Error Rate Problem
    Repeated tests for significant relationships, if uncorrected for the number of tests, can artifactually inflate statistical significance
  6. SCV: Unreliability of Measures
    Measurement error weakens the relationship between two variables and strengthens or weakens the relationship among three or more variables.
  7. SCV: Restriction of Range
    Reduced range on a variable usually weakens the relationship between it and another variable.
  8. SCV: Unreliability of Treatment Implementation
    If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.
  9. SCV: Extraneous Variance in the Experimental Setting
    Some features of an experimental setting may inflate error, making detection of an effect more difficult.
  10. SCV: Heterogeneity of Units
    Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult.
  11. SCV: Inaccurate Effect Size Estimation
    Some statistics systematically overestimate or under estimate the size of an effect.
  12. Ways to increase statistical power
    • use matching, stratifying, or blocking
    • measure, and correct for covariates
    • use larger sample sizes
    • use equal cell sample sizes
    • improve measurement (e.g. increase range of measurements/reduce dichotomized variables, add additional waves of measurement) 
    • increase the strength of treatment (e.g. increase dose differential)
    • Use a within-participants design
    • Use homogeneous participants selected to be responsive to treatment
    • Reduce random setting irrelevancies
    • Ensure that powerful statistical tests are used and their assumptions are met. (e.g. transforming the data)
  13. Internal Validity Definition
    Inferences about whether observed variation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured.
  14. Threats to Construct Validity
    • Inadequate Explication of Constructs
    • Construct Confounding
    • Mono-Operation Bias
    • Mono-Method Bias
    • Confounding Constructs with Levels of Constructs
    • Treatment Sensitive Factorial Structure
    • Reactive Self-Report Changes
    • Reactivity to the Experimental Situation
    • Experimenter Expediencies
    • Novelty and Disruption Effects
    • Compensatory Equalization
    • Compensatory Rivalry
    • Resentful Demoralization
    • Treatment Diffusion
Card Set
HPEB 818
for the quals