Is Audio Science Review going about it all wrong? Or partly wrong? Or all right?

[SmarterthanU]

Because of a century and more of their use in sensory science. And in audio, specifically, in how they manage to demonstrate great human sensitivity to changes in level, frequency response, localization, polar pattern, and data compression.

If you want to argue that null results for the "magic things" that engineers and scientists can't measure indicate test issues, go for it. We call that "special pleading." If you can't hear it without peeking, you can't hear it.

You cannot say for sure that any test of perception is accurate as to definition as every person's perception is inherently biased to their physicality/biology and experience/memory. Hence why we see so much disagreement in audio hardware.

 

[SIY]

Frankly, I can't make any sense out of that.

If you can't hear something without peeking, you can't hear it. If you can hear something without peeking, you can hear it. It's really not that complicated.

 

[SmarterthanU]

It is complicated as you dont know what is being measured exactly. This test could just be measuring a person's (lack of) ability to remember what they "heard" to enough detail to be able to compare/contrast with accuracy to what they are "hearing". This is very likely the case as the "hearing" will bias what was "heard" as most people will confuse the two "hearings" rapidly after the first note or two.

 

[SmarterthanU]

These test most likely only prove that people, in general, do not have very good auditory memory which seems very reasonable given that most people do not have very good memory at all in any capacity. More than half of the population does not have enough memory retention to even attend college let alone finish high school with decent grades.

 

[SmarterthanU]

Another way to clarify is to examine photo memory tests where people are given a picture to see for a set time and then have to remember/recall any detail a moment later. Most people fail miserably at this test even when given a lot of time to take in the picture. So why should these audio test be any different?

 

[SmarterthanU]

What would even confuse the subject even more is to present another photo similar to the first but different slightly and to ask what is different between the two photos. Most would remember wrong or only remember one or two things that stood out to them that were biased to their previous memories and experiences.

 

[Blumlein 88]

Another way to clarify is to examine photo memory tests where people are given a picture to see for a set time and then have to remember/recall any detail a moment later. Most people fail miserably at this test even when given a lot of time to take in the picture. So why should these audio test be any different?

Had a teacher do something like this when in the 8th grade. She was a new teacher, and was showing us interesting things about thinking/memory she said. So she had a desk drawer with 26 items in it. She let each student go up and look until they thought they knew what was in it. Most looked for a couple minutes. I got my turn just as the assistant principal came in the door to give the teacher a message. I glanced at it for 3 seconds and sat down. The teacher turned to see I wasn't there and asked had I taken long enough to look at the items in the drawer. I informed her I had. She asked was I sure. I confirmed that I knew what was in the drawer.

She then asked us to write down what was in the drawer. I was the only person to get all 26 items. One other student got 20 items. I think a couple more were near that. Most managed around half. I was blessed with a good memory. I didn't try to remember the items by name, I merely held the image in mind. It actually was more the physical relationship between the items. A pattern to match.

But that isn't a test of visual acuity. It is a test of memory. A test of visual memory.

One could do a test of aural memory in other ways. And it wouldn't be a test of aural acuity either.

Echoic memory comes into play. It has been tested with representative groups. So it isn't that we need some rarities to know how it works in groups of people. Shorter is better at least into the range of a few seconds, and an average group of people without hearing issues has a quite reliable echoic memory for use in testing hearing acuity. The results of testing taking that into account matches in broad parameters what we know physically about how hearing works. So there is plenty to conclude unsighted testing isn't just a test about memory. If nothing else you can go from positive results of various sound phenomena to the point you get negative results. You don't need the highest performing group for this to work reliably. Just a representative group of enough size and with no hearing pathology.

 

[watchnerd]

And this thing costs 800 € for just one channel!

Compared to high end audio, that's cheap for tubes + solid state.

 

[SmarterthanU]

Had a teacher do something like this when in the 8th grade. She was a new teacher, and was showing us interesting things about thinking/memory she said. So she had a desk drawer with 26 items in it. She let each student go up and look until they thought they knew what was in it. Most looked for a couple minutes. I got my turn just as the assistant principal came in the door to give the teacher a message. I glanced at it for 3 seconds and sat down. The teacher turned to see I wasn't there and asked had I taken long enough to look at the items in the drawer. I informed her I had. She asked was I sure. I confirmed that I knew what was in the drawer.

She then asked us to write down what was in the drawer. I was the only person to get all 26 items. One other student got 20 items. I think a couple more were near that. Most managed around half. I was blessed with a good memory. I didn't try to remember the items by name, I merely held the image in mind. It actually was more the physical relationship between the items. A pattern to match.

But that isn't a test of visual acuity. It is a test of memory. A test of visual memory.

One could do a test of aural memory in other ways. And it wouldn't be a test of aural acuity either.

Echoic memory comes into play. It has been tested with representative groups. So it isn't that we need some rarities to know how it works in groups of people. Shorter is better at least into the range of a few seconds, and an average group of people without hearing issues has a quite reliable echoic memory for use in testing hearing acuity. The results of testing taking that into account matches in broad parameters what we know physically about how hearing works. So there is plenty to conclude unsighted testing isn't just a test about memory. If nothing else you can go from positive results of various sound phenomena to the point you get negative results. You don't need the highest performing group for this to work reliably. Just a representative group of enough size and with no hearing pathology.

A more apt example would be a video of similar length to an audio sample. Still reasonable to me to assume these test are limited as the differences between samples is slight enough for people to not be able to remember exactly enough with enough detail to accurately compare. If a test shows most people cannot hear the difference between two samples then how can you be sure this is not due to memory issues? You cannot be 100 percent sure so these test are not as valid as they are made out to be in proving what is audible or not.

 

[Shadrach]

What would even confuse the subject even more is to present another photo similar to the first but different slightly and to ask what is different between the two photos. Most would remember wrong or only remember one or two things that stood out to them that were biased to their previous memories and experiences.

Unfortunately audiophiles haven't in most published ABX tests, set up the tests properly.
With any test the first thing should be to prove the test. This then avoids the arguments such as you put forward regarding memory.
It does make the test a little more time consuming. First test with something where the differences can be heard. We chose an amp playing at high volume that distorted and then clipped at that volume, level matched to an amp that didn't. Done properly, it's a lot of fun. Everybody gets quite excited because they score a positive result. As the volume decreases it gets more and more difficult. First the obvious clipping falls away and then the distortion.
There are lots of variations; tones can be added to a music file, certain frequencies can be altered etc etc. The point of it all is at the end of the test, having scored with decrasing accuracy rather than just null scores, ime people accept the test works and accept the results more easily.

 

[SmarterthanU]

Had a teacher do something like this when in the 8th grade. She was a new teacher, and was showing us interesting things about thinking/memory she said. So she had a desk drawer with 26 items in it. She let each student go up and look until they thought they knew what was in it. Most looked for a couple minutes. I got my turn just as the assistant principal came in the door to give the teacher a message. I glanced at it for 3 seconds and sat down. The teacher turned to see I wasn't there and asked had I taken long enough to look at the items in the drawer. I informed her I had. She asked was I sure. I confirmed that I knew what was in the drawer.

She then asked us to write down what was in the drawer. I was the only person to get all 26 items. One other student got 20 items. I think a couple more were near that. Most managed around half. I was blessed with a good memory. I didn't try to remember the items by name, I merely held the image in mind. It actually was more the physical relationship between the items. A pattern to match.

But that isn't a test of visual acuity. It is a test of memory. A test of visual memory.

One could do a test of aural memory in other ways. And it wouldn't be a test of aural acuity either.

Echoic memory comes into play. It has been tested with representative groups. So it isn't that we need some rarities to know how it works in groups of people. Shorter is better at least into the range of a few seconds, and an average group of people without hearing issues has a quite reliable echoic memory for use in testing hearing acuity. The results of testing taking that into account matches in broad parameters what we know physically about how hearing works. So there is plenty to conclude unsighted testing isn't just a test about memory. If nothing else you can go from positive results of various sound phenomena to the point you get negative results. You don't need the highest performing group for this to work reliably. Just a representative group of enough size

A great example that would be appropriate would be when TVs were advancing in resolution faster than content. Artifacts normally hidden on 480p TVs were clear to me at 1080i as I was into the technology and trained myself to see the artifacts. I had to point out the artifacts to other people who still did not see what I was talking about until I rewound the video many times and only after great effort on my part did people finally see what was very clear to me.

So you cannot draw conclusion that people could not see the artifacting as it was visible but just not perceived accurately by most people. If they do not perceive correctly they cannot remember correctly.

An even more apt example would be the very slight difference between 1080p upsampled to 4k and native 4k content. Although the fact that people could not even see huge pixelated dots on a 1080 screen should be evidence enough that perception is not reliable and therefore memory is unreliable for these tests.

 

[Shadrach]

A great example that would be appropriate would be when TVs were advancing in resolution faster than content. Artifacts normally hidden on 480p TVs were clear to me at 1080i as I was into the technology and trained myself to see the artifacts. I had to point out the artifacts to other people who still did not see what I was talking about until I rewound the video many times and only after great effort on my part did people finally see what was very clear to me.

So you cannot draw conclusion that people could not see the artifacting as it was visible but just not perceived accurately by most people. If they do not perceive correctly they cannot remember correctly.

An even more apt example would be the very slight difference between 1080p upsampled to 4k and native 4k content. Although the fact that people could not even see huge pixelated dots on a 1080 screen should be evidence enough that perception is not reliable and therefore memory is unreliable for these tests.

If as you state, memory is so unreliable, you couldn't tell the difference between one unit and another under any conditions which means you may as well choose by some other criteria. measurements would seem the most obvious answer.

 

[andreasmaaan]

These test most likely only prove that people, in general, do not have very good auditory memory which seems very reasonable given that most people do not have very good memory at all in any capacity. More than half of the population does not have enough memory retention to even attend college let alone finish high school with decent grades.

As humans we are unable to listen to two audio systems/streams simultaneously and discern between them (for example, if we want to try to discern a high-res file from an mp3, playing both files simultaneously will prevent us from hearing them as separate sources; we must listen to the files one after another to have any chance of discerning between them).

Therefore, whenever we compare two audio systems - whether we do this sighted or unsighted - we must rely on memory.

It is well-established that our long-term auditory is much less reliable than our echoic memory, which lasts at most a few seconds.

Therefore, if we are unable to distinguish between two different audio inputs based on echoic memory - as is the case in fast-switching ABX testing - then we are unable to distinguish between them under any circumstances whatsoever.

 

[SmarterthanU]

As humans we are unable to listen to two audio systems/streams simultaneously and discern between them (for example, if we want to try to discern a high-res file from an mp3, playing both files simultaneously will prevent us from hearing them as separate sources; we must listen to the files one after another to have any chance of discerning between them).

Therefore, whenever we compare two audio systems - whether we do this sighted or unsighted - we must rely on memory.

It is well-established that our long-term auditory is much less reliable than our echoic memory, which lasts at most a few seconds.

Therefore, if we are unable to distinguish between two different audio inputs based on echoic memory - as is the case in fast-switching ABX testing - then we are unable to distinguish between them under any circumstances whatsoever.

Doesnt perception alter what people remember in any case as what is perceived is biased from our experience and biology? Is it not the case that our brains must filter out what our consciousness cannot process due to inherent limitations in processing speed? This filtering would then make any memory inconsistent and personally biased which is the case based on common experience.

 

[andreasmaaan]

Doesnt perception alter what people remember in any case as what is perceived is biased from our experience and biology? Is it not the case that our brains must filter out what our consciousness cannot process due to inherent limitations in processing speed? This filtering would then make any memory inconsistent and personally biased which is the case based on common experience.

Of course. This is an inherent limitation of our memory system. Couple that with:

(1) the fact that long-term auditory memory is even less reliable than echoic (very short-term) auditory memory

(2) the inherent limitation of human consciousness that I mentioned in my previous post - i.e. that we can only have one conscious experience at one time

and you have precisely the reason why the most reliable way to compare two auditory experiences is via echoic memory.

Moreover, the most effective way to prevent personal bias from infecting our memories is by making any experiment double-blind.

 

[SmarterthanU]

Of course. This is an inherent limitation of our memory system. Couple that with:

(1) the fact that long-term auditory memory is even less reliable than echoic (very short-term) auditory memory

(2) the inherent limitation of human consciousness that I mentioned in my previous post - i.e. that we can only have one conscious experience at one time

and you have precisely the reason why the most reliable way to compare two auditory experiences is via echoic memory.

Moreover, the most effective way to prevent personal bias from infecting our memories is by making any experiment double-blind.

Even if a test shows that 99.999% of the people tested cannot distinguish between two audio samples it is still possible there are perceptible differences that can be heard if our focus is directed appropriately. So that would render double blind test inconclusive as well, no?

 

[andreasmaaan]

Even if a test shows that 99.999% of the people tested cannot distinguish between two audio samples it is still possible there are perceptible differences that can be heard if our focus is directed appropriately. So that would render double blind test inconclusive as well, no?

All science is by definition inconclusive, sure. The best we can ever say is that, under the specific test conditions, no subject was able to discern a difference.

Science can't offer conclusions; it can only offer hypotheses that, even under the most rigorous test conditions, cannot be falsified.

Take an example like genetics. If this theory is correct, then the following hypothesis should be true: "Offspring will be genetically similar to their parent(s)."

We can then genetically test a number of parents and offspring. If our tests are sufficiently rigorous, we should not find any example of offspring that are not genetically similar to their parents. If we do the tests and indeed find that all tested offspring are genetically similar to their parents in such a way as the theory would predict, we don't say that the theory of genetics is conclusively proven. Rather, we say that it is an explanation that cannot be falsified. If it is our only explanation that cannot be falsified, we say it's the best explanation.

 

[SIY]

Science can't offer conclusions; it can only offer hypotheses that, even under the most rigorous test conditions, cannot be falsified.

Typo? The whole point of science is the ability to test hypotheses, i.e., any scientific hypothesis is falsifiable. If it cannot be falsified, it's not science.

 

[andreasmaaan]

Typo? The whole point of science is the ability to test hypotheses, i.e., any scientific hypothesis is falsifiable. If it cannot be falsified, it's not science.

Misundestanding (or misuse haha?) of the word "cannot". Of course a scientific hypothesis must be falsifiable, but a strong hypothesis will not be falsified by experimentation.

 

[DonH56]

Even if a test shows that 99.999% of the people tested cannot distinguish between two audio samples it is still possible there are perceptible differences that can be heard if our focus is directed appropriately. So that would render double blind test inconclusive as well, no?

One can always argue based on the need for an infinite sample to cover all possible outcomes. One could easily argue the sun will not come up tomorrow and at some point will probably be correct.

Interesting choice of username. I should change mine to "StupiderthanU" just to be consistent. DBTs are not my day job but I'll throw out a couple of observations (from fallible memory). Consider the following IME/IMO.

Auditory memory is about 6 seconds IIRC (and there's my out) from ages-ago AES studies. Amir can probably cite references. Switching time in any sort of comparison test must be well below that to provide valid results. The idea is to listen for differences, not that A or B (or X) is better. Some claim differences can only be clear over extended listening tests; the evidence says the opposite because memory is fallible, plus our perception of the music is influenced by our mental and physical state. The latter is a big player in differences, of course -- e.g. "mood matters". What long-term listening does IMO is help provide additional examples and things to listen for. Many times when I thought I heard something better or just different, a return to the previous system showed it was there all along. Getting something different/new tends to heighten our focus as listeners. In effect, we train ourselves to be better listeners, so we can identify changes. The (a) catch with long-term testing is that we really need to go back to the original state whenever a difference is detected, and that can be very hard to do. Not only does it often take considerable time, but often is flat-out impossible because we've gotten rid of the previous component(s). Thus the need for short-term testing like DBT/ABX to ensure what we think we hear is really there.

In the primordial past I was actually able to do that. It proved conclusively for me that some differences in electronics do exist, but the vast majority went away in a DBT. The new things I found were there all along. It was rather humbling for someone respected for his hearing and being able to distinguish the granular structure of atomic and molecular alignment in cables (OK, the latter might be a stretch...) to discover most things (cables) sound the same after all.