The McGurk Effect, Freaky auditory illusion Options

[Dologan]

I know this is probably not what was in mind with the "psychoacoustic phenomena" this forum is intended for, but this sure as hell has to count as one... (if not, feel free to move this wherever is most appropriate)

The McGurk effect -how our *vision* can dramatically affect what we hear- is an effect first described in 1976, but which was recently talked about on a Scientific American podcast. Check it out at SciAm or YouTube.

At least in my case, it is absolutely freaky how well it works and makes a wonderful example how our mind really has the last word on what we would think is up to our ears. Also makes you wonder how spot on this forum is in supporting ABX as a basis for any auditory claim...

This post has been edited by Dologan: May 13 2008, 00:54

[DigitalMan]

I remember a study that was a sighted speaker comparison where only the color of the grilles were changed. Yellow scored the best by a comfortable statistical margin - go figure. Placebo is extremely powerful.

[Axon]

That's actually a lot more subtle than I expected. The "da" sound is really ambiguous when looking at the face, and the "ba" sound is really ambiguous when I'm not. But yeah, the effect works.

[Polouess]

That's actually a lot more subtle than I expected. The "da" sound is really ambiguous when looking at the face, and the "ba" sound is really ambiguous when I'm not. But yeah, the effect works.

Dosn't work at all for me

[retro83]

On a somewhat related note, I was linked to this MP3 from one of the related items at the end of that video.
Listen to it with headphones on.

I've never heard such realistic spatial sound, especially in the vertical plane. Really is an odd sensation listening to it.

[Dologan]

That's actually a lot more subtle than I expected. The "da" sound is really ambiguous when looking at the face, and the "ba" sound is really ambiguous when I'm not. But yeah, the effect works.

Dosn't work at all for me

Really? Do you hear the same "ba" regardless of whether you are looking at the guy or not? That's interesting...
My girlfriend says she can't hear the D sound, she gets a mix of "ga" and "ba" sound while looking and "ba" when not, but she does perceive the effect.
In my case, while looking at the guy, I hear something between "Ga" and "Da", but most definitely NOT "Ba", no matter how much I concentrate on hearing the "B" sound. On the other hand, as soon as I look away or close my eyes, the "Ba" sound pops up, crystal clear. If I open and close my eyes during the video, I can even get to hear something like "ba da, da ba, ba da" or whatever, depending on the state of my eyelids. Really spooky.

[Dologan]

On a somewhat related note, I was linked to this MP3 from one of the related items at the end of that video.
Listen to it with headphones on.

I've never heard such realistic spatial sound, especially in the vertical plane. Really is an odd sensation listening to it.

Whoa, man, indeed! Even EAX has never had such a realistic spatial effect, although, to be fair, it's probably significantly easier to record it than to emulate it that way.

[Martel]

On a somewhat related note, I was linked to this MP3 from one of the related items at the end of that video.
Listen to it with headphones on.

I've never heard such realistic spatial sound, especially in the vertical plane. Really is an odd sensation listening to it.

Whoa, man, indeed! Even EAX has never had such a realistic spatial effect, although, to be fair, it's probably significantly easier to record it than to emulate it that way.

You would have to render the sound from the 3D model of the environment (but using sound refraction properties of materials instead of textures). Perhaps through some sort of sound raytracing (except bass which is not very ray-like).
Even if you created a reliable algorithm for such rendering, you're not likely to see it in discrete sound cards anytime soon since it would probably require more computational horsepower than you can put together on a chip.

[kdo]

On a somewhat related note, I was linked to this MP3 from one of the related items at the end of that video.
Listen to it with headphones on.

I've never heard such realistic spatial sound, especially in the vertical plane. Really is an odd sensation listening to it.

Is it anything different from binaural recordings?



( the "da da" trick works on me perfectly. )

[xmixahlx]

everyone realizes that this is crafted, right? it uses video editing to mux a different audio track onto a video track...

it is fun tho


later

[Fandango]

In my case, while looking at the guy, I hear something between "Ga" and "Da", but most definitely NOT "Ba", no matter how much I concentrate on hearing the "B" sound. On the other hand, as soon as I look away or close my eyes, the "Ba" sound pops up, crystal clear. If I open and close my eyes during the video, I can even get to hear something like "ba da, da ba, ba da" or whatever, depending on the state of my eyelids.

Same here. I think it means our visual sense is very dominant over the auditory sense.

everyone realizes that this is crafted, right? it uses video editing to mux a different audio track onto a video track...

Sure, why not. You can call it an audio-visual illusion. You can (re-)create Illusions artificially so that the phenomena can become noticable by the participant, because many illusions occur in nature without us noticing them. Isolating such an illusion, will make it easier or even possible to notice the paradox in perception.

[Fandango]

PS: I just found out about this lady... http://en.wikipedia.org/wiki/Diana_Deutsch
I'd like to have one of her CDs!

[Dologan]

Same here. I think it means our visual sense is very dominant over the auditory sense.

Well, not necessarily. It can also refer to how much vision intervenes in speech recognition, which is a critical ability humans have had to evolve; apparently to such extent that it is able to override auditory input.

everyone realizes that this is crafted, right? it uses video editing to mux a different audio track onto a video track...

Yeah, that's the whole point of it! To illustrate how in case of conflicting information, what you see can trump, or half-trump (with odd results) what you actually hear.

[raintheory]

On a somewhat related note, I was linked to this MP3 from one of the related items at the end of that video.
Listen to it with headphones on.

I've never heard such realistic spatial sound, especially in the vertical plane. Really is an odd sensation listening to it.

I think this can be accomplished by using binaural stereo mics with a flat plate seperating them (simulating a head?), and moving the sound back and forth... Then using a lowpass filter to simulate the sound of it going behind the head, and a highpass filter for in front of the head... Would this work?

I am intrigued by this, I do a fair amount with experimental sound works. Might have to try to see if I can replicate it using this method.

[Dologan]

I think this can be accomplished by using binaural stereo mics with a flat plate seperating them (simulating a head?), and moving the sound back and forth... Then using a lowpass filter to simulate the sound of it going behind the head, and a highpass filter for in front of the head... Would this work?

I am intrigued by this, I do a fair amount with experimental sound works. Might have to try to see if I can replicate it using this method.

I am not sure a flat plate would really be necessary to provide left-right directional localization; proper separation of the microphones could probably do the trick alone, although a head simulation would no doubt help. What I am not sure is how exactly sounds could be made as being in front or behind you. In human hearing, I'd guess the shape of the outer ears plays an important role in it. Why would you say the lowpass/highpass filter could provide that effect? Even trickier is probably the elevation localization, at which humans aren't very good anyway. Outer ear/echo effects, perhaps?

EDIT: Ah, instead of just blindly speculating, we should have just checked the wikipedia first Dummy head recording

This post has been edited by Dologan: May 15 2008, 11:05