CHFD 5110 Exam 2

The process of observing and recording the observations that are collected as part of a research effort

Are you measuring what you inteded to measure?

• Under the umbrella of construct validity
• Focuses on where the operationalization (ie- measure) is a good translation of the construct

• On it's face, does the operationalization look like a good translation of the construct?
• Does it seem like it fits for this?

Operationalization is checked against the relevant content domain for the construct

• The performance of your operationalization (ie- measure) is checked against some other criterion
• A prediction of how the operationalization will perform on some other measure based on your theory or construct

• Under criterion-related validity
• Operationalization's ability to predict something it should theoretically be able to predict
• ie- Is your SAT score able to predict your success in college?

• Under criterion-related validity
• Operationalization's ability to distinguish between groups that it should theoretically bea ble to distinguish between
• ie- Can a measure of depression distinguish between people who are depressed and those who aren't?

• Under criterion-related validity
• Degree to which the operationalization is simuilar to other operationalizations to which it should theoretically be similar
• You'll see this one the most
• ie- Does your measure of depression correlate highly with another measure of depression?
• You want high correlations, but not TOO high :)

• Under criterion-related validity
• Degree to which the operationalization is not similar to other operationalizations to which it theoretically shouldn't be similar to
• ie- Do you like your therapist & How well you cope in your family... they SHOULDN'T correlate
• You want low correlations

• Threat to construct validity
• A really long way to say that people didn't do their jobs. They didn't define what they were doing well enough

• Threat to construct validity
• Pertains to treatment or program
• Used only one version of the treatment or progam
• ie- Trying only one dosage of a drug in a drug trial instead of trying multiple dosages

• Threat to construct validity
• Pertains to the measures or outcomes
• Only used one type of measure instead of several
• This is a problem because of accuracy- we need multiple ways to look at results!!

• Threat to construct validity
• People guess the hypothesis and respond to it rather than respond naturally
• This could cause you to mislabel the "cause." You'll attribute effect to treatment rather than good guessing

• Threat to construct validity
• Ppl make themselves look good just because they're in a study

• Threat to construct validity
• Can bias consciously or unconsciously

• Threat to construct validity
• Conclude that the treatement has no effect when it's only that level of the treatment which has none (might need more or less to see a result)

• Threat to construct validity
• Ppl get more than 1 treatment
• Labeling issue
• ie- Don't only get tutoring from your program, also get help from family at home

• Threat to construct validity
• Does the testing itself make the groups more sensitive or receptive to the treatment?
• Labeling issue

• Threat to construct validity
• You didn't measure your outcomes completely
• You didn't measure some key affected constructs at all
• ie- Viagra was supposed to help with blood pressure

• Consistency or repeatability of a measure
• Getting the same result every time you do the study
• Based on True Score Theory of Measurement

• Observed score = True ability + Random error
• We never really have the true score!

• Caused by any factors that randomly effect measurement of a variable across the sample
• Things that happen to you on a daily basis but don't change the dynamics of the entire group
• ie- One person is having a bad day the day you administer a depression measure

• Caused by factors that systematically affect measurement of a variable across the sample
• Does affect the group average
• ie- Temperature in the room

There's an observed score, a true score, and an error

• Pilot test the measures
• Train interviewers/observers
• Double-check data
• Statistically adjust for error
• Use multiple measures of the same construct

• Reliabilty ranges from 0 to 1
• It's the percet of the true score that is there
• Shows how much of a true score we have and how much error we have

• One type of reliability
• Assesses the degree to which different raters/observers give consistent estimates of the same phenomenon about the estimate of the true score
• Percent agreement
• Usually the cut-off for reliability is .80 (about 80% accurate)

• One type of reliability
• Correlation btw 2 observations on the same test administered to the same (or similar) sample on 2 diff occasions
• Time btw observations is crucial
• Looking for stability over time

• One type of reliability
• Correlation btw to observations on parallel forms of a test administered to the same sample
• 2 forms of the "test" can be used independently but it requires that you generate a lot of items (b/c you need to create 2 diff versions)
• Assumes randomly divided halves are equivalent
• Looking for stability across forms

• One type of reliability
• Single test administered to a sample on one occasion
• Assesses the consistency of the results for diff items for the same construct w/in the measure
• Average item-total correlation
• How consistent is the measure inside itself?
• Probably see Conbach's alpha (a) the most (the average of all possible split half correlations)

• Part of internal consistency reliability
• The average of all possible split-half correlations

• Both are important, but aren't related
• You can have one without the other
• You want it to be reliable AND valid if you can

• Numerical values simply name the attribute
• No ordering of values is implied (the number is a place holder, not an indication of how good or bad something is)
• Numerical values are simply "short codes" for longer names
• ie- "1" for male and "2" for female

• Attributes can be rank-ordered
• Distances/intervals between attributes have no meaning (you can rank who's higher and who's lower, but you don't know the gap between them)
• ie- Degrees of agreement (strongly agree/agree/neutral...)

• Attributes can be rank-ordered
• Distances/intervals between attributes do have meaning
• ie- Temperature on Fahrenheit scales- if it's 65 today and 75 tomorrow, we can say tomorrow will be 10 degrees warmer
• Ratios don't make sense (80* isn't twice as much as 40*)
• No meaninful Absolute Zero (0* is not the absense of heat)

• Always a meaninful Absolute zero
• ie- Anything you count or Income in dollars
• Ratios make sense (If you made $2 and I made $4, I made $2 more than you did)

• Frequently used method in our field
• Questionnaires & interviews

• Dichotomous response format (ie- "yes" or "no")
• Questions based on level of measurement (ie- rank order of preferences)
• Filter or contingency questions (1st question determines qualification to answer the next one)

• First question determines qualification or necessary experience to answer the next question
• ie- "If no, skip to question 4."

Questions with more than one response

• Fill-in-the-Blank
• Check the Answer
• Circle the Answer

• Written text
• ie- "Tell me more about your experience with..."

• When you combine 2+ variables to reflect a more general construct
• ie- SES Index

• Conceptualization
• Operationalization & Measurement
• Development of rules for calculating the score
• Validation of the index score
• CODV!!
• Consider- does it match what I'm trying to measure??**

• Take qualitative ideas (like willingness to have immigrants in your country) and come up with a way to measure them quantitatively
• Where a lot of error occurs
• Typically yields a single numerical score that represents the construct of interest
• ie- Degree of depression
• Is a process, not a response scale... so it requires more development
• Seen more than indexing

• Hypothesis Testing: Is the construct or concept a single dimensional one?
• Exploration: What dimensions underlie some ratings?
• Scoring: For assigning values to responses

• Higher/Lower
• ie- Age, Height, GPA

• Constructs must be related
• ie- Shared activity & communication to measure marital satisfaction

• Unidimensional scaling methods
• Summative scale
• Scaling process, not response-format
• Start w/ large set of items you think all reflect the same construct
• Have a group of judges rate the items on how much they think it relates to the overall concept

• Likert scaling
• Throw out items with the total (summed) score across all items

• Likert scaling
• For each item, get the average rating for the top 1/4 of judges and the bottom 1/4
• Better discrimination

• If it's not numbers, it's qualitative
• Consists of words
• Used to generate new theories/hypotheses, Achieve deep understanding of an issue, Develop detailed stories to describe a phenomenon
• More about understanding something, not proving something

Consists of numbers

• A big umbrella, kind of like a theory
• Includes: Ethonography, Phenomenology, Field research, and Grounded theory

• Qualitative tradition
• Study within the context of a culture
• What only, no interaction
• Concerned with cause & effect
• Sometimes called "naturalistic research"
• Used mostly in anthropological research
• Most common approach is participant observation
• No limits of what will be observed and no real ending point

• Qualitative tradition
• Interested in the phenomenon from the perspective of the participants
• Acknowledge that objectivity is impossible to ascertain and often incl. a section in their report about themselves to acknowledge biases

• Qualitative tradition
• Researcher observes a phenomenon in its natural state (in situ)
• An observation in the environment where they'd normally be
• No implemented controls or experimental conditions to speak of (nature is emphasized over culture here!)
• Only concerned with observing. Not concerned with cause & effect
• Especially useful in observing social phenomena over time

• Qualitative tradition
• To develop a theory, grounded in observation, about a phenomenon of interest
• Build a theory from the ground up or go in with preconceived notions
• Interview changes as you learn things
• Eventually get to a place where you've fully explored the phenomenon & have an idea what's going on

Think of these not as the umbrella, but as separate umbrellas under tha main umbrella of traditions

• Qualitative Method
• Researcher becomes a participant in the culture being observed
• Takes some time

• Qualitative Method
• Researcher not a member of the culture being studied but remains unobtrusive
• Just observing- not getting involved
• Don't get as much info as you would if you were participating

• Qualitative Method
• Direct interaction btw. the researcher and respondent
• No set direction- just kind of see where it goes
• There's always a little bit of structure, though, since you're interviewing this person for a reason

• Qualitative Method
• Intensive study of a specific individual or specific context

• Internal validity
• External validity
• Reliability
• Objectivity

• Credibility
• Transferability
• Dependability
• Confirmability

Est. that the results are credible from the perspective of the participant

Degree to which results can be generalized to other contexts

• Description by the researcher of changes w/in the context & how these might affect conclusions

Degree to which others can confirm or corroborate the results

• Unobtrusive measure
• The researcher collects data w/out the participant being aware of it

• Unobtrusive measure
• Systematic analysis of text in order to identify patterns

When you use data that's already been collected (ie- census report) for new research