HPEB 818

• Construct
• External
• Internal
• Statistical Conclusion

• Low Statistical Power
• Violated Assumptions of Statistical Tests
• Fishing and the Error Rate Problem
• Unreliability of Measures
• Restriction of Range
• Unreliability of Treatment Implementation
• Extraneous Variance in the Experimental Setting
• Heterogeneity of Units
• Inaccurate Effect Size Estimation

An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.

Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance of an effect.

Repeated tests for significant relationships, if uncorrected for the number of tests, can artifactually inflate statistical significance

Measurement error weakens the relationship between two variables and strengthens or weakens the relationship among three or more variables.

Reduced range on a variable usually weakens the relationship between it and another variable.

If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.

Some features of an experimental setting may inflate error, making detection of an effect more difficult.

Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult.

Some statistics systematically overestimate or under estimate the size of an effect.

• use matching, stratifying, or blocking
• measure, and correct for covariates
• use larger sample sizes
• use equal cell sample sizes
• improve measurement (e.g. increase range of measurements/reduce dichotomized variables, add additional waves of measurement) 
• increase the strength of treatment (e.g. increase dose differential)
• Use a within-participants design
• Use homogeneous participants selected to be responsive to treatment
• Reduce random setting irrelevancies
• Ensure that powerful statistical tests are used and their assumptions are met. (e.g. transforming the data)

Inferences about whether observed variation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured.

• Inadequate Explication of Constructs
• Construct Confounding
• Mono-Operation Bias
• Mono-Method Bias
• Confounding Constructs with Levels of Constructs
• Treatment Sensitive Factorial Structure
• Reactive Self-Report Changes
• Reactivity to the Experimental Situation
• Experimenter Expediencies
• Novelty and Disruption Effects
• Compensatory Equalization
• Compensatory Rivalry
• Resentful Demoralization
• Treatment Diffusion