Perceptual Differences Between Sighted and Unsighted Music Listening

[Anton S]

What perceptual differences do you notice when listening to music sighted versus unsighted, i.e. eyes open vs. closed? I have long contended that humans are primarily visual creatures. At least I am. Given a bimodal or multimodal sensory experience that includes both auditory and visual inputs, each will affect the other to a certain extent, but the visual generally contributes most to my experience.

This phenomenon is apparent whenever I relax and listen to music in a focused manner. It’s a very different experience for me when my eyes are open than when they are closed. With them open, I see my front equipment arranged a plane flanking my screen and their exact locations across my field of view. Although my music sounds fine in this case, the perceived front-to-back stage depth seems somewhat shallow, because my eyes are processing that flat plane of equipment. Also, left-right staging appears a little narrow, because my L/R speakers are relatively close together compared to my listening distance, and I can visually locate each component in front of me very precisely. Conversely, with my eyes closed and the visual input removed, left-right staging appears to broaden, and perceived front-back depth is enhanced, because I’m not seeing that flat plane of equipment before me. These perceptual changes do not occur immediately or all at once, because my memory of the visual stimulus lingers for some time, but after a short interval and gradually.

Curious about this phenomenon, I did some poking around and discovered several studies that looked at mixed-modality sensory inputs. The two cited below were generated from two very different perspectives, but drew congruous conclusions. The first, from the National Library of Medicine, discusses mixed-modal inputs from a medical perspective, and the second, from the US Army Aeromedical Research Laboratory, investigates the phenomenon with respect to HMD’s (Helmet Mounted Displays). However, both detail the “ventriloquism effect” and the impact of visual stimuli on audition, which I believe correlates well with the perceptual changes that I described above.

Interactions of Auditory and Visual Stimuli in Space and Time​

AUDITORY-VISUAL INTERACTIONS

I believe this phenomenon is reasonably universal, and that every sighted human with normal hearing perceives similar changes so some degree. If you’ve previously noted this phenomenon yourself, please share below. Unfortunately, if not and you’ve read this far, it’s probably too late for you to offer an unbiased assessment. Future perception will likely be influenced by expectation.

I’ve noticed another mixed modal sensory phenomenon, which I haven’t yet researched at all, but thought I’d mention in passing, just to see if anyone else has previously noted it. This one involves a physical/tactile element, rather than a visual one, in addition to the auditory one. I’ve worn glasses for many years – so long that I am unaware of the fact most of the time. However, I often remove them when relaxing to music, since there’s really no need for visual input. But as soon as I remove my glasses, the auditory experience immediately changes, even when my eyes are completely closed. The music sounds clearer and more immediate to me. The closest analogy I can think of is the difference between looking at a room through a window and being in the room. Weird, huh?

Although, as I said above, I haven’t investigated this phenomenon at all, I suspect that there are both physical and psychoacoustic aspects to it. First, my lenses and frames undoubtedly alter the sound reaching my ears. And second, the removal of constant pressure on my temples and the bridge of my nose alters my perspective by removing that tactile input, eliminating a subconscious distraction from the auditory experience. Has anyone else who wears glasses ever noticed a similar change?

 

[Jim Taylor]

When I watch a video of a performance, whether it's pop or classical orchestra, the audio sounds "better". I believe that is because a greater number of my senses are feeding the brain information, and the brain is busier processing that information, making the experience more engaging.

However, that doesn't mean that I'm going to listen to recordings any less often.

 

[olieb]

When I watch a video of a performance, whether it's pop or classical orchestra, the audio sounds "better".

When I close the eyes during a performance (or a video) my (strong) impression is that I hear more and better (distraction?).
Therefore I close my eyes most of the time, in live performances and even in the opera to some extent. For me that is much more engaging.

EDIT: Maybe it's already clear, but at home, listening to recordings, I have the eyes closed, I do not look at my gear or speakers. I just open them from time to time to check whether the world is still there.

 

[Anton S]

When I watch a video of a performance, whether it's pop or classical orchestra, the audio sounds "better". I believe that is because a greater number of my senses are feeding the brain information, and the brain is busier processing that information, making the experience more engaging.

However, that doesn't mean that I'm going to listen to recordings any less often.

When I close the eyes during a performance (or a video) my (strong) impression is that I hear more and better (distraction?).
Therefore I close my eyes most of the time, in live performances and even in the opera to some extent. For me that is much more engaging.

Two diametrically opposed perspectives right off the bat? Imagine that.

My perception is exactly the same as @olieb described, and I also often close my eyes at live venues to better enjoy the auditory experience. But Jim asserts that the audio sounds "better" to him when accompanied by a visual component.

Things that make you say, "Hmm."

 

[DVDdoug]

I really don't hear a difference. I assume it's my practical-rational personality... I also don't get a "realistic soundstage illusion" and the phantom center is somewhat vague. ( It's different with surround sound a center-channel speaker.) I really don't pay that much attention to it, except I enjoy stereo more than mono, and surround even more!

Sometimes I've enjoyed headphones in the dark. But I don't listen to headphones that much and it's been awhile since I've listened in the dark.

I've got a shelf-full of video concerts and I always watch the video, except that I've made MP3s of most of them for the car.

And since I was a kid, I've built sound activated lighting effects... Since the "color organs" in the 70's. In my living-room I have a giant ceiling-height "VU" meter which is similar to a regular LED VU meter but with the LEDs spaced about 4-inches apart. (It also has some other chasing/sequencing/blinking effects.) I don't always have the lighting effects on, but sometimes I enjoy the music & lights together.

In all of these cases, the visual (or darkness) doesn't affect the "sound" but it might affect my enjoyment or my mood.

 

[Anton S]

I really don't hear a difference. I assume it's my practical-rational personality... I also don't get a "realistic soundstage illusion" and the phantom center is somewhat vague. ( It's different with surround sound a center-channel speaker.) I really don't pay that much attention to it, except I enjoy stereo more than mono, and surround even more!

Sometimes I've enjoyed headphones in the dark. But I don't listen to headphones that much and it's been awhile since I've listened in the dark.

I've got a shelf-full of video concerts and I always watch the video, except that I've made MP3s of most of them for the car.

And since I was a kid, I've built sound activated lighting effects... Since the "color organs" in the 70's. In my living-room I have a giant ceiling-height "VU" meter which is similar to a regular LED VU meter but with the LEDs spaced about 4-inches apart. (It also has some other chasing/sequencing/blinking effects.) I don't always have the lighting effects on, but sometimes I enjoy the music & lights together.

In all of these cases, the visual (or darkness) doesn't affect the "sound" but it might affect my enjoyment or my mood.

A third country heard from. This one - "No-Perceived-Sound-Difference" - appears to be located midway between the countries of Sounds-Better-Sighted and Sounds-Better-Unsighted. My goodness!

I've never enjoyed headphones, because the audio has always sounded to me as if it's coming from inside my cabeza. I hate that, so I use headphones only when forced to do so, such as when flying. I do prefer multichannel audio, even upmixed stereo, over 2-channel or 2.1, so we're aligned on that front.

Also, like you, I have a considerable number of DVD and Blu-ray music video performances that I enjoy occasionally, but more often without the video playing than with it.

 

[thewas]

Two diametrically opposed perspectives right off the bat? Imagine that.

I would rather say two approaches which I can both familiarise with, for less conscious listening I also enjoy more when I see a video but for more a conscious one I usually close my eyes.

 

[Anton S]

I would rather say two approaches which I can both familiarise with, for less conscious listening I also enjoy more when I see a video but for more a conscious one I usually close my eyes.

Fair enough. When not fully engaged in the listening experience, I think anything goes. However, I believe I did qualify that my perception and preference were when listening in a focused manner.

 

[Chrispy]

My music listening came without video options for a great chunk of my listening experiences (recordings). Only in relatively recent years have I really combined my audio/video systems let alone having significant content in my library that has both. Looking at the gear doesn't do much for me particularly either. If I am watching a concert video it hasn't particularly occurred to me to turn off the visual. I'm probably more in the camp it doesn't matter a lot. Altho with music alone if comfy enough I might close my eyes but not really sharpening the auditory experience but rather just being comfortable (usually lying down rather than sitting too).

As to glasses, been wearing contacts for a very long time but I can see somewhat taking glasses off is just more comfy too.

 

[Anton S]

My music listening came without video options for a great chunk of my listening experiences (recordings). Only in relatively recent years have I really combined my audio/video systems let alone having significant content in my library that has both. Looking at the gear doesn't do much for me particularly either. If I am watching a concert video it hasn't particularly occurred to me to turn off the visual. I'm probably more in the camp it doesn't matter a lot. Altho with music alone if comfy enough I might close my eyes but not really sharpening the auditory experience but rather just being comfortable (usually lying down rather than sitting too).

As to glasses, been wearing contacts for a very long time but I can see somewhat taking glasses off is just more comfy too.

I suspect one's affinity for having a visual component in his/her entertainment may be established early in life. Generations X, Z and A were all likely to have been subjected from birth to television productions as surrogate nannies. Probably most of the Millenials, as well. I'm an old Boomer. We never even had a TV in our home until I was ten, and even after we did, my mom strictly rationed my time in front of the "Boob Tube." When I wasn't playing outside, most of our family leisure time was spent listening to radio broadcasts, so I didn't form a strong attachment to video when I was a child.

However, the intended central theme of my OP, remains as stated in one of the papers that I linked in that post, which is as follows: "At times a visual object will capture the location of the auditory object and bias it towards the visual object even without them actually being perceived as a fused object." In other words, one's perception of stage depth and instrument placement when enjoying music is significantly altered by visual stimuli.

 

[Chrispy]

I suspect one's affinity for having a visual component in his/her entertainment may be established early in life. Generations X, Z and A were all likely to have been subjected from birth to television productions as surrogate nannies. Probably most of the Millenials, as well. I'm an old Boomer. We never even had a TV in our home until I was ten, and even after we did, my mom strictly rationed my time in front of the "Boob Tube." When I wasn't playing outside, most of our family leisure time was spent listening to radio broadcasts, so I didn't form a strong attachment to video when I was a child.

However, the intended central theme of my OP, remains as stated in one of the papers that I linked in that post, which is as follows: "At times a visual object will capture the location of the auditory object and bias it towards the visual object even without them actually being perceived as a fused object." In other words, one's perception of stage depth and instrument placement when enjoying music is significantly altered by visual stimuli.

I might disagree with that conclusion, but "at times" is vague in any case. I've also watched a lot of visuals with music that aren't related to the performers particularly let alone some "video object". YMMV.

 

[Anton S]

I might disagree with that conclusion, but "at times" is vague in any case. I've also watched a lot of visuals with music that aren't related to the performers particularly let alone some "video object". YMMV.

Although the ventriloquism effect is a well-documented and broadly accepted phenomenon, as is the corollary in the cited passage, opposing viewpoints are, of course, always welcome. But the courtesy of a reference to at least one paper supporting that opinion would be greatly appreciated. Just basically saying, "I think you're wrong" seems a bit thin.

 

[olieb]

In other words, one's perception of stage depth and instrument placement when enjoying music is significantly altered by visual stimuli.

I would absolutely subscribe to this. I found
But for the quote

At times a visual object will capture the location of the auditory object and bias it towards the visual object even without them actually being perceived as a fused object."

I do not even understand what it means. What would be an example of an object capturing a sound stream without the brain fusing the two?

In relation to the perception of music and the localisation of sound I found Griesinger interesting and helpful.
Here is a presentation

http://www.davidgriesinger.com/Localization,%20Loudness%20and%20Proximity%208.pptx

from his homepage

http://davidgriesinger.com/

From slide 8 (presentation above) with binaural examples (that are supposed to listen to with individually equalised headphones):

With eyes open the sound in both seats is perceived as similar. But the brain knows the difference. The sound in row K is less exciting, less engaging, and easier to forget.

He is more interested in the effects with eyes closed as (for him as for me) the visual component tends to remove the auditive differences. But he has data and research to back that up.

I am actually a visual person, therefore a video WILL distract me, but most of the time it is "interesting" at best and rarely brings anything to the table that is equally strong for me as the music (the kind I am listening to).

One last point. In relation to the combined perception of music and visual there is the phenomenon of synaesthesia as it is reported from i.e. Olivier Messiaen. He "saw" music as colors, chromesthesia

Chromesthesia - Wikipedia

en.wikipedia.org

 

[Jim Taylor]

I am actually a visual person, therefore a video WILL distract me, but most of the time it is "interesting" at best and rarely brings anything to the table that is equally strong for me as the music (the kind I am listening to).

If I had used the phrase "more interesting" instead of the word "better" in my post (the second post, above), it would have been more accurate to what I was trying to say. If I attend a symphony orchestra, I always try to get a seat where I can watch the musicians ... especially the string section.
I suppose the ultimate expression of this "mental function"would be ballet or opera. For me, listening to "the Magic Flute" or "The Nutcracker" is wonderful ... but not nearly as wonderful (or engaging) as being seated in the audience for a performance, where I can see the activity on stage.

Hope this clarifies what I said.

 

[Old_School_Brad]

I don't experience it any differently whether my eyes are open or closed. However, I do think differently, and on certain tracks or in certain moods, I find that closing my eyes helps me relax more. It’s about savoring the moment, really. I imagine that’s pretty common for most people.

 

[Anton S]

What would be an example of an object capturing a sound stream without the brain fusing the two?

My apologies. I now realize that I should have quoted the entire paragraph from that study to better clarify. Here is the full paragraph:

"At times a visual object will capture the location of the auditory object and bias it towards the visual object even without them actually being perceived as a fused object. For example, Bertelson and Radeau (1981) reported that the attraction of auditory localization towards visual objects may occur even when fusion is not present, that is, the stimuli are not correlated. Thus the visual capture may depend strongly on the synchrony of auditory and visual stimulations and not necessarily on the realism of the auditory-visual pair (Radeau and Bertelson, 1977). The extent of the visual capture depends on the distance between the locations of the visual and auditory stimuli and is the strongest around the midline (Hairston, Wallace, Vaughan, Stein, Norris, and Schirillo, 2003)."

The underline is mine.

EDIT: Text from the same paper just prior to that paragraph ...

"Capture effect
"Another important consideration when designing or choosing an audio system for an HMD is to realize that information received in one sensory channel can be affected by information received through another channel. This phenomenon is called the capture effect. One of the most familiar examples of this phenomenon is the ventriloquism effect (VE) (Howard and Templeton, 1966). The VE refers to our tendency to perceive sounds as coming from the same location as a visual event, as would be the case of perceiving the sound as coming from the ventriloquist’s dummy. In this case, the location of a visual object, the dummy, captures the perceived location of the sound source, the ventriloquist. Thomas (1941) describes the tendency for listener judgments of sound source location to be biased in the direction of a flickering light, especially if the light is in sync with the sound. The perceived location can either be fused with the visual source, or shifted towards the source.
"The effect is strong and compelling for smaller angles of 20° to 30°. Thurlow and Jack (1973) report that it is greatly decreased at 60° (but still occurred at least some of the time for 6 out of 10 participants). Although Thurlow and Rosenthal (1976) observed some capture at 170°, it is probable that this is due to the human tendency to confuse the auditory location of sounds near 0° and 180° (see Chapter 12, Visual Perceptual Conflicts and Illusions).
"In the case of the ventriloquist, the percept of the sound source location is fused with the apparent visual source of the sound (Figure 14-3). Cognitive factors affect the strength of the VE by increasing the likelihood that the visual and auditory sources will be fused (Radeau and Bertelson, 1977). For example, researchers have varied the apparent probability that the visual object is the source of the sound, using video monitors, puppets and stationary objects as the visual targets (Thurlow and Jack, 1973; Warren, Welch, and McCarthy, 1981) (Figure 14-4). As might be expected, the sound is more likely to be fused with the visual object if the visual object appears to be a probable source of the sound."

 

[Thomas Savage]

Iv always found my cheap JBLs have the remarkable ability to sync to folks talking on the TV. That's likely me seeing something and ' forcing ' the localised sound image to a degree.

Going on that maybe some human heads being put in the appropriate place behind the speakers might be beneficial for some albums .

It's the assumptions that alter how you perceive sound ime , often I'm completely unaware of it .

Taking away my sight , in the dark or closing my eyes helps focus my listening experience, giving me a enhanced ' picture ' of the music.

I always think back to when I was a child , we had a some kind of spinning thing on the dining room table that if i concentrated I could get to spin the other way . Similarly I could and still can convince myself I'm at the other end of the bed when I wake up. So when I open my eyes I'm disorientated.

The mind boggles .

 

[Elkerton]

I was smiled at by Janet Baker at Carnegie Hall when I opened my eyes after a song she sang.

There was a violinist in a chamber music ensemble whom I hated watching while she played. I closed my eyes and her performance improved immeasurably.

 

[Chrispy]

Although the ventriloquism effect is a well-documented and broadly accepted phenomenon, as is the corollary in the cited passage, opposing viewpoints are, of course, always welcome. But the courtesy of a reference to at least one paper supporting that opinion would be greatly appreciated. Just basically saying, "I think you're wrong" seems a bit thin.

I may have taken the comment too broadly or at least missed how your opinion particularly is a fact.

 

[Pareto Pragmatic]

My perception is exactly the same as @olieb described, and I also often close my eyes at live venues to better enjoy the auditory experience. But Jim asserts that the audio sounds "better" to him when accompanied by a visual component.

Things that make you say, "Hmm."

The live performance/video adds a factor compared to just listening with open/closed eyes. You can see the sounds being generated, you can anticipate a cymbal crash, you can see how many people are singing.

For me, I think I would hear more with eyes closed for music I knew, but more with eyes open for music that is new to me. Not that I am going to test it or anything but that's my guess.