We __ events of our world through our sensory __. Events give rise to __ in different sensory channels; the brain stiches this __ __ together to __ events.
- multisensory information
Give 2 reasons why multisensory processing improves perceptual estimates?
- 1. info from one sensory modality can be ambiguous by itself but clarified and constrained by info from another modality (eg. Sekuler et al, 1997, bouncing ball illusion)
- 2. all sensory processing subject to noise (eg. photon noise, stochastic nature of action potentials and loss of info at higher coginitive levels). Having multiple modalities helps minimise variability.
Talk to me a bit about the modularity of the sensory systems. Why is it important?
- Sensory processing is, broadly speaking, initially modular
- followed by subsequenst stages of comination
- Important that noise is independent from these different modalities, because we gain a lot more from the cancelling out that happens with multisensory processing
- Basic organisation of cerebral cortex suggests division of labour and modularity (eg. visual cortex, auditory cortex)
- HOWEVER: some debate as to the extent to which this mulsisensory processing happens
- More evidence (Ghazanfar & Schroeder, 2006) suggests multisensory processing in almost all cortical areas.
Does one sensory modality dominate others? Esp in relation to sound and vision.
- Some examples.
- Vision: ventriloquist effect - movement of puppet's mouth leads viewers to infer the source of sound
- Sound: perception of number of visual events dominated by number of auditory events such as beeps and flashes (Shams et al, 2000)
- Fiarly widely-held idea - certain modalities have different dominances because they are intrinsically beteer at signalling certain types of info.eg. Sound - temporal resolution; vision - spatial resolution (about 100:1 to sound)
Describe what Shams et al (2000) experiment was about.
- the two-flash illusion
- participant presented with single flash of light accompanied by two sound beeps
- they perceive two flashes
However, even if these intrinsic advantages exist, does this mean some modalities should 'capture' certain tasks?
- Computationalluy, it does not sensible to believe this happens
- eg. echoey cave - shouldn't rely on sound as much
- Does not make much sense to throw away one source of information
Instead of dominance, how else can the brain process two or more different sensory modalities?
- eg. McGurk effect
- film of a person visually saying /ga/ with soundtrack of person saying /ba/ is heard as /da/.
- Thus, interestingly, their perceptual interpretation of the event is something does not correspond to either modality in isolation.
- But what actually governs when modalities are combined and when single modalities appear to dominate?
Estimates of the stimulus get more __ (have lower __) when sensory modalities are __. However, instead of simple averaging of the different modalities, what is done that is more effective?
- The estimate is the combination fo the singals weighted by the reliability of each.
What do we typically use to describe sensory estimates? What can it tell us?
- Gaussian model (Normal Distribution)
- Can tell us the variance of many different responses from the same modality to see what its reliability is like.
How would you combine the information from two modalities using Gaussian distributions?
- statistically optimal inference using Maximum Likelihood Estimation - of weighted combination and minimising variance
- Multiply the two distributions together
- Resultant estimate - Gaussian that is centred between the two component estimates and has lower variance (higher reliability) than either modality alone.
- mean of this estimate is weighted average of the means of the individual cues (where weight is determined by reliability of each cue).
Why is the Maximum Likelihood Estimation (MLE) useful for understanding how different modelities are used?
- It seems to resolve the problem of dominance vs combination arguments
- Experimentally, results which seemed to show dominance may have just been due to the fact that it was very difficult to seprated a combined estimate form a single componeent (when one modality's reliability is very high)
Decribe the classic study in 2002 which provided evidence for optimal integration.
- Ernst & Banks (2002)
- Studied perception of object size from visual and haptic cues
- asked to judge thickness of a horizontal bar based on 3 info:
- 1. visual
- 2. haptic (graping it)
- 3. visual and haptic
- Asked to compare standard object with various others and to judge smaller or larger
- Participants better using visual cues, but crucially, visual+haptic yielded best performance.
- Change of condition:
- add visual 'noise' - by randomising poisition of some of the elemets that defined object's size
- Result: As predicted by MLE model, as reliability of visual cue was reduced, participants moved to give more weight to the haptic cues. Visual dominance --> haptic dominance
Describe the follow-up study to Ernst & Banks (2002) study.
- Alais and Burr (2004)
- Manipulation of azimuth of visual and auditory targets ('is this left or right of a reference?')
- Vision: spots of light on computer
- Sound: positions defined by interaural time difference (ITD)
- Judgements on sound alone much worse than vision (needed bigger differences from reference).
- When both cues combined for same location, performance was better, again, adding weight to MLE model.
- Condition:When sound and vision presented together but specifed different locations
- --> Judgements of position pulled all the way towards the visual spatial position
- Ventriloquist effect
- Condition 2:Blur the visual target (making it less reliable)
- --> judgements shifted twards being dominate dby auditory cue.
Final word about MLE model and combination of multiple senses.
- This MLE model has helped the understanding of multisensory combination away from the ad hoc way of understanding the way sensory info were combined.
- Evidence from Ernst & Banks; Alais & Burr as well as many subsequent studies show that brain uses a process similar to the 'statistically optimal' thing to do.
How is Bayes theory/rule important in understanding MLE and the combination of mulsisensory information?
- Bayes rule states that an unknown probability of an events can be calculated using the knowledge of related probabilities
- This can be thought of as a broader throty in which MLE sits
- Basically: the use of previous knowledge of the statistical likelihood of events when making inferences (eg about position or shape of object).
- Classic example of this: interpretation of shading patterns
- shading patterns can look like a hole or bump, but humans interpret shading patterns by assuming the pattern was produced by illumination above the object.
We can think about __ __ in the exact same way as we thought about sensory cues. It will have a __ __ and can be __ with sensory data using the same rules for integrating different sensory modalities (__ __ __).
- prior information/knowledge
- probability distribution
- multiplying probability distributions