GrovesChapter8

Can respondents and interviewers, if they are used, complete the questionnaire easily and as they were intended to?)

Subject matter experts review the questions to assess whether their content is appropriate for measuing the intended concepts, or in which questionnaire design experts assess whether the questions meet the three standards for questions.

Based on a technique called Protocol Analysis--where subjects think aloud as they work on the problems and their verbalizations are recorded.

Tape recording pretest interviews, then making systematic observations of how questions are read and answered, can also provide useful information about the question.

Experimentally compare different methods of data collection, different field procedures, or different versions of the questions.

Another name for randomized experiments.

The extent to which the survey measure accurately reflects the intended construct.

• Estimating Validity with data external to the survey
• Estimating Validity with Multiple indicators of the same construct

Comparing survey responses to existing records.

The opposite of reliability. When a survey has high reliability for a population, then it has low simple response variance.

Repeated interviews to measure the same constructs.

Reliability as in high reliability for a population (or low simple response variance), then the index of inconsistency which equals (1-reliability)

The gross difference rate, which for a dichotomous variable is merely twice the simple response variance.

A widely used measure of reliability of multi-item indices.

Do the questions ask about the right things?

Do respondents understand the questions consistently; do they have the information required to answer them; are they willing and able to formulate answers to the question