Reliability and Validity

• An underlying trait that is reesponsible for some observeable behavior
• Indirect measure

• Refers to inferences made from measured variables to theoretical constructs
• Examines how well the assessment “matches up with” the construct

—“the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of the test”

• Construct Underrepresentation (Less)
• —Construct Irrelevant Variance (More)

• Not measuring the construct as broadly as intended
• “the degree to which a test fails to capture important aspects of the construct”*

—Measuring consistently something that is not part of the construct

• Evidence based on Test Content
• Evidence based on Response Processes
• Evidence based on Internal Structure
• Evidence based on Relations to Other Variables
• Evidence based on Consequences of Testing

• Analysis of the relationship between the test’s content and the construct of interest
• Themes, wording, and format of the items, tasks, or questions on a test, as well as the procedural guidelines for its administration and scoring

• Analyses of the response processes of examinees is used to determine the fit between the construct and  the examinee’s actual performance or response
• Includes the processes of judges, raters, or observers when evaluating examinee’s performances .

• Use logical or empirical analysis
• —Most frequently relies of “expert” judgment
• Part of test development

• Theoretical and empirical
• Process focused on a study of examinees
•      Interview test takers
•      Think alouds
• Examine rating or scoring process

The degree to which the relationship among test items and test components conform to the construct which the interpretations are based.

• Item analysis
• Reliability
• Factor analysis

• relationship between the test scores and variables outside of the test
• ie: GRE predicts student success in graduate school
• Relationship to other tests

• Convergent and discriminant evidence
• Test criterion evidence (predicts and a function of purpose)
• Validity Generalization (Can the instrument be generalized to other situations)

Relationship between scores on measures of the same construct

Relationship between test scores and measures of different constructs, should not be related

• Intended consequences happen
• Unintended consequences are not occuring especially issues of bias, fairness

• Intended
• Shaping the curriculum
• Teaching the content of the test
• Unintended Consequences
• Narrowing the curiculu
• Teaching to the test and only the test
• Subpopulation differences
• Develops over time

The extent to which a cause and effect relationship is isolated from competing influences

• Determines the soundness of conclusions/interpretations of a causal relationship
• Are the variables influenced by different variables

• Time threats
• Group threats
• Mortality
• Atypical behavior

Impacts on dependent variable over time are different because of factors other than the treatment variable

• Explanations for changes in the variables other than experimental differences created by the researcher
• Selection of the group may cause

• Loss of subjects in a study
• Impacts all studies long enough to have dropouts

• Research design cannot eliminate
• Actions of those in the groups which change the treatment (Treatment and control group interact about the intervention)

• The effect of the first test on the second
• Effect of a publication about the treatment
• Solutions:
• Lengthen time interval between tests
• Disguise use of prestest (don't disclose next test)
• Use control groups

• An event that occurs during treatment that affects subject response
• Usually events that could be controlled
• Solutions
• Use control groups
• Use shorter time

• Naturally occuring process within participants that occur because of time and may change their performance
• Inc. fatigue, boredom, growth, intellectual development
• Solutions:
• Shorten the time of the study
• Use a control group with a similar maturation rate

• Changes in measurement procedure in a pretest - posttest study
• Inc: calibration, rater changes, score use
• Solutions:
• control group
• standardize the measurement procedure

• One group with post-test only such as pilot testing
• Or
• One group with pre-and post test
• These are weakest designs because they do not strongly link group changes to treeatment

• Designs that involve a control group but do not use random selection
• Or
• One experimental group with multiple tests, the first is a baseline

• Time threats
• Selection (group changes at time of intervention)
• Instrumentation
• Maturation

• Pre-test, post test, control group design w/random assignment
• Post test control group design with random assignment
• Still has threats to validity but usually eliminates group threats

• Use random assignment
• Use Pretest
• Use control/comparison group

• Within-subject designs
• Between subjects designs
• Mixed designs

• Pre-experimental designs
• Quasi experimental designs
• Experimental design

• Measures individuals within a group multiple times both before and after treatment
• Change over time is of interest
• Applies to pre-experimental and quasi experimental

• Compares the scores of two or more groups
• Group comparison are of interest
• Applies to quasi-experimental and experimental designs

• Both between and within designs used
• ie: look at individual student scores as a result of a treatment in two classes
• Applies to quasi-experimental and experimental

• Statistical adjustment based on theoretical argument
• Matching of variables between groups
• Random assignment creates random equivalence on all variables
• Matching and Random Assignment

• Regression towards the mean
• Selection
• Selection by time interactions
•      selection by maturation
•      selection by history

• Applies to pre-post design
• Scores (both high and low) regress to the mean
• Inflates low group and deflates high group change
• Impacts in gain score situations with extreme groups

• Avoid comparison of extreme groups if possible
• Retest-retest to establish a more "stable" baseline
• Use a control group for each extreme
• Use high reliability measures

• A threat that is due to the different group characteristics present at the start of the study
• Affects all quasi-experimental studies

• Internal validity threats interact with selection to produce effects not due to treatment
• Selection maturation (mature at different rates)
• Selection history (groups from different settings so different events)

• Random assignment
• Matching (only relates to variables being matched)
• Random assignment and matching
• Check pretest equivalence applies to quasi and true experimental design (ANOVAs testing for this)

• subjects leave the study for different/systematic non-random reasons
• results in selection artifact because posttest group is different

• Control groups if mortality cause is the same
• Shorter time interval between start and finish
• Monitor incidence of mortality
• Use pretest to compare scores of those who dropped and those who did not

• not considered part of validity by all because cannot be controlled by design
• Differences between groups not caused by the design
• Caused by group communication or public knowledge of treatment groups

• NOT conrolled by random assignment
• Monitor the research project
• If possible prevent groups from communicating with each other

look at threats to the sample which prevents generalizing to the population of interest

Complete set of observations about which we draw conclusions

the subset of the population from which the sample is drawn

Actual observations included in the study

• 1. Define the observation unit (individual or object)
• 2. Define the target population
• 3. Define the boundaries of the population (affects generalizability)
• 4.Define the sampling technique
• 5. Obtain a sampling frame to use with the technique
• 6. Select the sample

• Probability samples
•      Simple random, Systematic, stratified, cluster
• Non-probability samples:
•      Convenience, quota, purposive

• Every item has an equal chance of being selected
• Has no control on sample make-up

• Select every kth element of sample
• Starting point should be randomly selected

• Separate samples are created for each strata (characteristic of interest)
• Ensures each characteristic of interest is represented
• Can be weighted in relation to population

• The population is divided into heterogeneous clusters
• Clusters should be already formed in the population
• One cluster is randomly selected

• Two step procedure after clusters are identified
• 1. Random selection of a cluster
• 2.  Random sample in each cluster, often based on strata

• Random or stratified random (can underrepresent larger units)
• Probability Proportionate to size (units are weighted based on size of unit)

• Non-probabalistic sampling
• Based on easily accessible elements in a population
• Subject pools
• Volunteers
• Often subject to all external validity threats

• Non-probabalistic sampling
• Like stratified without random element
• Begin with a matrix targeting characteristics
• Collect data from each person having the characteristics
• Tries to represent the population

• Non-probabalistic sampling
• Used for pilot testing or manipulation
• Use when members of subset are easy to identify but hard to include all
• Use when only generalizing to that subset

• The difference between the true score of the sample and the true score of the population
• Can be estimated with probalistic sampling methods
• NOT an error made in creating the sample

• A listing of the population from which a sample will be drawn
• As representative population is formed, consider if groups are not represented how results will be impacted

• Bias results from non-response if they are different than responders
• Is there a systematic difference between those that did not participate and those that did?
• Can be accounted for through statistical modeling

• How well do your findings generalize across settings, samples and times?
• How can we generalize to the population on the basis of sample-based results?

• Interaction of selection and treatment
• Interaction of setting and treatment
• Interaction of history and treatment

• Occurs when treatment effects only generalize to those selected in the same way as the sample
• Applies to non-probalistic sampling
• Solutions
• random selection
• make participation convenient

• Treatment effects generalize only to settings used in the study
• Solution:
• Vary the settings and analyze IV/DV within settings

• Look at when you collected your sample, will it generalize across other time frames
• Solution:
• Replicate the study at different times
• Conduct research review to see if prior evidence refutes the relationship

Sampling error is the discrepency between a sample statistic and the corresponding population parameter.

• Null: What is assumed to be true about a situation at the start of an experiment
• Alternative: is true is null is false, usually research is trying to prove null is no longer true because of treatment

• Null hypothesis is true but you reject it and conclude the alternative is true
• You are concluding treatment was effective when it was not

• The null hypothesis is false but you decide not to reject it.
• You conclude the treatment was ineffective when it was

• The probability of making a Type I error
• Most common is .05 (needs to be lower is loss of income, job, health, life is at stake)
• Defines an unlikely sample assuming a null hypothesis is true

• The probability you correctly reject a false null hypothesis
• or
• We reject null when alternative is true
• (1-BETA)

• Increase the following to increase power
• sample size
• significance level (but inc significance makes errors more likely)
• effect size
• Increasing variance DECREASES power