-
What are the 3 reasons for assessing IOA?
Consistency of measurement
Minimizes observer bias
reflects the quality of operational definition
-
What is the difference between agreement between observers and accuracy? Is it possible to have one but not the
other? Explain.
Agreement - how well data from independent observers correspond
Accuracy - How well the data reflects what actually happens
Yes it is possible, two people can make mistakes in the same way therefore you can have agreement but not accuracy
definition could be too broad, thus you have accuracy but not agreement
-
What is the difference between and reliability and validity? Can you have one
without the other? Explain.
Reliability - consistent measurement of the construct
Validity - measurement reflects the construct
- yes. you can measure something reliably and it not reflect the construct
- and you can measure a construct inaccurately
-
Is accuracy and validity the same? Why or why not?
No, you can accurately measure a bx that has nothing to do with the overall construct
measuring eye contact for reading comprehension
-
Complete the following sentence using the terms “accurate” and “reliable.” If data
are _________, they must be ___________. Why is the converse not true?
Give an example.
- accurate, reliable
- the converse is not true because two people can take data inaccurately in the same way which would make it reliable, but not accurate
-
Do you have to check IOA during each phase of a study? Why or why not?
yes, IOA should be assessed in at least every phase of the experiment to ensure consistency
-
Be able to calculate IOA using the frequency ratio method, Point-by-Point
Agreement Ratio method (scoring agreements as occurrences and non-occurrences), and
frequency-within-interval method.
FREQ RATIO - smaller total/larger total X 100
Point By Point - Agreements/agreements +Disagreements X 100
Freq within interval ? pg 193
-
What does “chance” refer to? How might
one handle expected “chance” levels of agreement?
level of agreement expected by randomly marking occurences
compute IOA using agreements on nonoccurences
-
In addition to “chance,” there are 4 other sources of bias that can obscure interpretation of interobserver agreement.
Describe each source of bias.
reactivity of reliability assessment - bx changes when you know your work is being checked
observer drift - change in the application of definitions of bx over time
Expectancies and Feedback - when observers expectations of change and feedback from experimenters combine, it may influence data taken by observer
Complexities of observations - the less complex the observations, the higher the agreement
-
Define reactivity and observer drift.
Reactivity - changes the people make when their bx is being monitored or evaluated
Observer Drift - change in the application of definitions of bx over time
-
Given that interobserver agreement provides no information about either accuracy
or reliability, why do you suspect that this is the most common approach to evaluating measurement procedures and data?
requires the least amount of response effort and is the best representation of the construct of accuracy and reliablity
-
When using an interval agreement procedure for calculating IOA, explain how the
percent agreement can be inflated by especially frequent or infrequent
responding.
especially frequent refers to that fact that agreement will be high because of the number of intervals in which the behavior occurs. If it is occurring a lot high agreement may happen
INfrequent responding referes to agreements based on non-occurrence which would be the same for low freq bxs but in reverse.
|
|
