Master Thread: Are measurements Everything or Nothing?

[AdrianusG]

I walked into audio stores and would lust after something just because of the brand and what it looked like. Made me assume it was always a step up. I think I bought at least 4 different premium CD players in the 90s. And integrated amplifiers. And speakers. And honestly I am not sure how much of a progression it really was.

"I walked into audio stores and would lust after something just because of the brand and what it looked like"

Ditto
for instance Accuphase and Luxman, didn't have the the money for it though, still haven't , oh well, perhaps in my next life then

 

[welwynnick]

I do and I inquired, and so did many, many others. Science is clear, without controlled testing, you can’t trust perceptions. If you want to inquire further, please do and publish the results of your controlled test.

I wrote about this in 2009 (when I had reasonable hearing - I don't now - nothing superhuman here).

I used to be an active member of AVForums, swapped gear with friends, and did lots of blind group listening tests at local dealers. On 26 Jan 2009 ( a day I remember well) I got my hands on a BDP and AVP that could both do HD codec decoding. I'd become quite obsessed with jitter at the time, and expected that processor decoding would sound better than player decoding due to the different digital audio replay architecture (how the audio clock is regenerated in the processor). Conventional wisdom was that they should sound the same because bits were bits, but I thought I knew better than everyone else, and several users had already claimed to hear a (sighted) improvement with processor decoding. I wanted to prove that I understood what made HiFi tick, and also to prove that I had golden ears. I could hardly wait to listen and confirm my convictions for myself. The anticipation had built up for some time, and I wonder if you can guess what happened next? I played a Bluray with a particularly musical HD audio soundtrack with the player configured to perform the decoding and output LPCM audio, and listened as closely as I could several times. Then I set the player to output bitstream, and listened with great expectation. Hard as I tried, I could hardly hear the slightest difference between them. I posted on AVF that "the difference was so small that I doubt I could tell the difference reliably". This was hugely disappointing and embarrassing, having firmly planted myself in the "processor decoding is better" camp. However, I owned up to the fact that I was wrong all along, or else I had cloth ears. After covering myself in shame (you can see this was a big deal to me) I went back to the lounge and looked at the Bluray case again. I hadn't realised that the BD soundtrack wasn't lossless compressed Dolby TrueHD or DTS-MA after all, but was in fact LPCM. Therefore the player had been outputting LPCM in both cases, so the audio replay architecture and hence the sound quality would have been the same. But I didn't figure this out until afterwards. During the comparison I thought I had been listening to two things that should have been technically and audibly different, so expectation bias was running riot. I knew what I was going to hear beforehand, but I didn't. Subsequently I played a different disc with a TrueHD soundtrack, and processor decoding did sound different. The point is that my comparison was not compromised by being sighted or biased. Where there was in fact no difference at all, that was what I heard. I can't speak for everyone else, but I think I did prove to myself that I don't imagine what I hear. The player was a Denon 3800 and the processor was an Onkyo SC886,. Other equipment combinations may have given a different result, and the test probably wouldn't work with more recent equipment.

Price list for new Onkyo PR-SC886

Right, some impressions. There's a great deal to say, and I won't be able to do it all at once. I've had the golden opportunity to answer a lot of questions, many of which relate to the Denon DVD 3800BD. For the purpose of this thread, though, I'll just consider it as a good HD MC audio...

www.avforums.com

 

[welwynnick]

"I walked into audio stores and would lust after something just because of the brand and what it looked like"

Ditto
for instance Accuphase and Luxman, didn't have the the money for it though, still haven't , oh well, perhaps in my next life then

Same here. I used to have a perfectly good Pioneer player, but replaced it with an expensive Denon that had exactly the same performance in my system. Pride of ownership, I suppose.

 

[pkane]

The point is that my comparison was not compromised by being sighted or biased.

You keep repeating this as if you could possibly know that this was true or not. Whether or not you were led astray by bias is not something you can decide by thinking about it or rationalizing. This can only be proven out by proper controlled testing, as is well documented by all the science behind perception.

 

[welwynnick]

You keep repeating this as if you could possibly know that this was true or not. Whether or not you were led astray by bias is not something you can decide by thinking about it or rationalizing. This can only be proven out by proper controlled testing, as is well documented by all the science behind perception.

It was a controlled test because I didn't know what I was listening to (despite my best efforts).
I can decide that I wasn't led astray by bias (enormously strong though it was) because I did not establish a difference or preference.
It was a null test, with a null result, which was the right result.
The result would only have been subject to bias if I had expressed a preference.

 

[pkane]

It was a controlled test because I didn't know what I was listening to (despite my best efforts).
I can decide that I wasn't led astray by bias (enormously strong though it was) because I did not establish a difference or preference.
It was a null test, with a null result, which was the right result.
The result would only have been subject to bias if I had expressed a preference.

That's the problem: if you see what you are listening to, or have any other clue as to which one that is, your test is not controlled. Cognitive bias doesn't work just in one direction, it works in both, you can hear a difference where there isn't one, and you can hear no difference where there is. Again, you can only speculate as to why you were perceiving (or not perceiving) differences.

 

[welwynnick]

That's the problem: if you see what you are listening to, or have any other clue as to which one that is, your test is not controlled.

My intention had been to make it a sighted and uncontrolled test, and then the result certainly would have been biased.
Had that been the case I'd have been happy, as it would have proved my academic theorising to be right. But then I wouldn't have mentioned it here.
As it happens I was miserable as hell, because either I was thick, or I was deaf, or both.
It was a dark day, but I admitted to my failure straight away, no hiding the truth, however painful.
As it turned out my test was controlled because I didn't know what I was listening to.

Cognitive bias doesn't work just in one direction, it works in both, you can hear a difference where there isn't one, and you can hear no difference where there is.

Well ain't that true! Everyone told me that I didn't like the Revel speakers because of my bias in favour of them!
But in this case it didn't work in either direction.

 

[pkane]

Well ain't that true! Everyone told me that I didn't like the Revel speakers because of my bias in favour of them!
But in this case it didn't work in either direction.

Anyone who tells you what "your bias" is going to tell you knows nothing about bias. It is not something easily predicted by anyone (including you) and regardless of the result of your test, if you didn't use proper controls, bias was almost certainly part of it. Whether or not it influenced your decision to like or not like something nobody can tell, including you. Perception tests can't be controlled "by accident". It takes careful planning and design.

 

[welwynnick]

Well ain't that true! Everyone told me that I didn't like the Revel speakers because of my bias in favour of them!

Anyone who tells you what "your bias" is going to tell you knows nothing about bias. It is not something easily predicted by anyone (including you) and regardless of the result of your test, if you didn't use proper controls, bias was almost certainly part of it. Whether or not it influenced your decision to like or not like something nobody can tell, including you. Perception tests can't be controlled "by accident". It takes careful planning and design.

The subject of the speaker dem only came up because I replied to Angsty's post:

“Believing is hearing. That's right; if you believe you will hear a difference, then there's a good chance that you will.” — Roger Russell

I was biased in favour of the Revels, but nonetheless I didn't like them, but you're saying that I didn't know I had a subconscious bias against them.
That's fine. Maybe it's true, or maybe it's speaker positioning. Either way I don't care, it's not important.

What's important to me is that bias didn't affect my processor decoding listening.
If there had been any bias, it wouldn't have mattered if it was conscious or not, I would have preferred one over the other because I believed that they were different.

 

[olieb]

What's important to me is that bias didn't affect my processor decoding listening. If there had been any bias, it wouldn't have mattered if it was conscious or not, I would have preferred one over the other because I believed that they were different.

Bias is not absent just because you somehow got something right one time.
Bias works like a dice, but a highly unfair one. Therefore the outcome is unpredictable, you don' even know how unfair it is.
Therefore you need blindfolding AND statistics to control it.
"Conscious bias" is a trivial kind of bias. If you consciously decide what you hear or not, you are just sabotaging the test. This too can be controlled by blind testing.

 

[welwynnick]

I can assure you that bias was very much present - I was absolutely determined to hear a difference that I didn't know didn't exist.

You could say that I was indeed consiously deciding what I heard or not, I was just sabotaging the test. But by failing the intended test, I had passed the unintended test.

I also did the test several times with the same result.

 

[Jim Taylor]

What's important to me is that bias didn't affect my processor decoding listening.
If there had been any bias, it wouldn't have mattered if it was conscious or not, I would have preferred one over the other because I believed that they were different.

Your stubborn refusal to try to understand information that has been given to you indicates that you are not here in good faith.

Jim

 

[Blumlein 88]

I can assure you that bias was very much present - I was absolutely determined to hear a difference that I didn't know didn't exist.

You could say that I was indeed consiously deciding what I heard or not, I was just sabotaging the test. But by failing the intended test, I had passed the unintended test.

I also did the test several times with the same result.

It is a hurdle you will need to get over to benefit from the knowledge of people on this forum. Differences have to be truly huge to be observed accurately in the kind of listening you are doing. Whether your results are positive, negative, match blind results or don't are not the point. The point is you are making judgements using a faulty methodology that is inherently unreliable. While it may have been accurate in some instances or not, you cannot depend upon it in subsequent instances in the future. I don't know what would convince you.

 

[welwynnick]

Your stubborn refusal to try to understand information that has been given to you indicates that you are not here in good faith.

Jim

I appreciate your considered response, but on the contrary, I can assure you that my background and principles have long been firmly rooted in science and engineering, and I am only here because I do believe in the value of objective measurements.

Yes, I am being deliberately stubborn, but that's because I won't accept that all subjective listening results are always wrong, which is often the message being thrust down people's throats in this thread in particular. Yes, they are often wrong, but not every single time. I want to prove that succumbing to bias isn't inevitable, and it can sometimes be overcome.

When I have time, I will get an ADC and DAC and software to make my own audio analyser, just to play with. I want to correlate what I hear and what I measure, which I'm very curious about. Where I can do blind testing, I will, but for now my hands are a bit full. It will happen, and I will write about what I find, but just not this week.

 

[DonR]

Yes, I am being deliberately stubborn, but that's because I won't accept that all subjective listening results are always wrong, which is often the message being thrust down people's throats in this thread in particular. Yes, they are often wrong, but not every single time. I want to prove that succumbing to bias isn't inevitable, and it can sometimes be overcome.

It is not a question of subjective listening being right or wrong but whether the results can be trusted. Since they are a product of both individual conscious and unconscious bias, logic dictates that these results cannot be trusted.

 

[olieb]

But by failing the intended test, I had passed the unintended test.

I also did the test several times with the same result.

There is no "several times" if the trials are not independent. And this they are not, if you repeat the very same thing sighted.
And so there is no "test" in the first place, at least not in any scientific meaning, not more so than accidentally predicting the throw of a coin is passing the test on being a clairvoyant and psychic.
You might have gotten it right, yes, but that is all that there is to it. You might be astonished, ok, but that is a feeling thing.

 

[BDWoody]

Yes, I am being deliberately stubborn, but that's because I won't accept that all subjective listening results are always wrong,

No one is saying that. They are saying it is unreliable, and that you seem to have a mistaken understanding of bias in general.

 

[welwynnick]

It is not a question of subjective listening being right or wrong but whether the results can be trusted. Since they are a product of both individual conscious and unconscious bias, logic dictates that these results cannot be trusted.

No argument there - they can't be trusted. Sometimes they're right and sometimes they're wrong. It's very difficult to know which, so therefore they can't be trusted.

 

[antcollinet]

Yes, I am being deliberately stubborn, but that's because I won't accept that all subjective listening results are always wrong,

You are not listening are you? I told you the first time back in January, then again, in March when you constructed this straw man that this is not what is said. It was linked at you again in the last few days.

Please stop it.

 

[solderdude]

I want to correlate what I hear and what I measure,

Good luck with that... (below certain thresholds)