272 U3

  1. What are the 3 reasons for assessing IOA?
    Consistency of measurement

    Minimizes observer bias

    reflects the quality of operational definition
  2. What is the difference between agreement between observers and accuracy?  Is it possible to have one but not the
    other?  Explain.
    Agreement - how well data from independent observers correspond

    Accuracy - How well the data reflects what actually happens

    Yes it is possible, two people can make mistakes in the same way therefore you can have agreement but not accuracy

    definition could be too broad, thus you have accuracy but not agreement
  3. What is the difference between and reliability and validity? Can you have one
    without the other? Explain.
    Reliability - consistent measurement of the construct

    Validity - measurement reflects the construct

    • yes. you can measure something reliably and it not reflect the construct
    • and you can measure a construct inaccurately
  4. Is accuracy and validity the same? Why or why not?
    No, you can accurately measure a bx that has nothing to do with the overall construct

    measuring eye contact for reading comprehension
  5. Complete the following sentence using the terms “accurate” and “reliable.” If data
    are _________, they must be ___________. Why is the converse not true?
    Give an example.
    • accurate, reliable
    • the converse is not true because two people can take data inaccurately in the same way which would make it reliable, but not accurate
  6. Do you have to check IOA during each phase of a study?  Why or why not?
    yes, IOA should be assessed in at least every phase of the experiment to ensure consistency
  7. Be able to calculate IOA using the frequency ratio method, Point-by-Point
    Agreement Ratio method (scoring agreements as occurrences and non-occurrences), and
    frequency-within-interval method.
    FREQ RATIO - smaller total/larger total X 100

    Point By Point - Agreements/agreements +Disagreements X 100

    Freq within interval ? pg 193
  8. What does “chance” refer to?  How might
    one handle expected “chance” levels of agreement?
    level of agreement expected by randomly marking occurences

    compute IOA using agreements on nonoccurences
  9. In addition to “chance,” there are 4 other sources of bias that can obscure interpretation of interobserver agreement. 
    Describe each source of bias.
    reactivity of reliability assessment - bx changes when you know your work is being checked

    observer drift - change in the application of definitions of bx over time

    Expectancies and Feedback - when observers expectations of change and feedback from experimenters combine, it may influence data taken by observer

    Complexities of observations - the less complex the observations, the higher the agreement
  10. Define reactivity and observer drift.
    Reactivity - changes the people make when their bx is being monitored or evaluated

    Observer Drift -  change in the application of definitions of bx over time
  11. Given that interobserver agreement provides no information about either accuracy
    or reliability, why do you suspect that this is the most common approach to evaluating measurement procedures and data?
    requires the least amount of response effort  and is the best representation of the construct of accuracy and reliablity
  12. When using an interval agreement procedure for calculating IOA, explain how the
    percent agreement can be inflated by especially frequent or infrequent
    especially frequent refers to that fact that agreement will be high because of the number of intervals in which the behavior occurs.  If it is occurring a lot high agreement may happen 

    INfrequent responding referes to agreements based on non-occurrence which would be the same for low freq bxs but in reverse.
Card Set
272 U3
272 U3