Master Thread: Are measurements Everything or Nothing?

[blackmetalboon]

Is it? Doesn’t knowing the results bias you? Because next time you know what to look for. You have been influenced by the test itself.

Doesn’t knowing what you should be looking for help to identify potential problems?

 

[voodooless]

@amirm, very curious what you guys did back in the day to speed up ABX testing of codecs even further?

How well did finding audible issues correlate with anomalies in null testing? And if the correlation was strong, did you then only ABX the parts that the null tests revealed? That sounds like a massive time saver? But maybe there was not enough correlation?

You could even do that in a modern ABX test. Just have people listen to the parts that are bound to have significant differences. Differences in DACs and Amps are much smaller than codecs obviously.

 

[birdog1960]

Is it really that complicated tho? If you hear an artist at a performance (or a recording replayed on the best measured available equipment) you loved and still love it just as much when you listen to a recording of it on lesser measured equipment without noticing differences that are audible to you, isn't that enough?

 

[bboris77]

It's not a measurement tool, it's a basic and absolutely necessary control to have any kind of experimental validity for determining audible factors.

What I meant to say is that I am not convinced that we can extrapolate any truly useful data from our listening experience even when imposing stringent measures on it like a rigorous blind-testing protocol. Sure, it would be possible to establish that a subject was able to distinguish signal A from signal B reliably. I just do not really see a purpose to it unless those signals measured identically using conventional measurement methods, which would indicate that something else needed to be measured to account for the difference between the signals.

 

[amirm]

No, you’d have different data: no audible differences if you don’t know where to look.

Nope. If I refuse to take the test, then that is that. As you plainly see, I did have the ability to find a difference. What you did then is draw the wrong inference regarding audible differences that existed. It would be proverbial putting the head in the sand.

By your logic, there should not be a seek bar, or ability create a segment loopback either. Why help them find differences, right?

You again not appreciating that difference is not constant across a clip.

 

[SIY]

I just do not really see a purpose to it unless those signals measured identically using conventional measurement methods, which would indicate that something else needed to be measured to account for the difference between the signals.

You may not see the purpose, but the purpose exists nonetheless- is any measured difference (or claimed phenomenon) actually audible? It's still a fundamental and necessary control for any sort of sensory evaluation for those that do these experiments. Not just audio, any sensory test requires controls. My own introduction to sensory testing was haptic, and double blind was an absolute necessity to get valid data. Ditto when I started doing organoleptic testing and research. Auditory is no different in that respect.

 

[amirm]

Sure, it would be possible to establish that a subject was able to distinguish signal A from signal B reliably. I just do not really see a purpose to it unless those signals measured identically using conventional measurement methods, which would indicate that something else needed to be measured to account for the difference between the signals.

Oh, there is great purpose in putting to question the assumption that there is no audible difference. In the test that Archimago put forward between 16 and 24 bit, I knew that during fade out the quantization steps could become audible. And audible they were. Folks saying it was impossible to tell the difference didn't think about that situation were full set of bits are not used. New insight was provided for people who had not appreciated the complexity of the inaudibility assertion.

 

[voodooless]

You again not appreciating that difference is not constant across a clip.

I’m not doing anything like that. I just proposed the ultimate help tool a few post above

 

[birdog1960]

You may not see the purpose, but the purpose exists nonetheless- is any measured difference (or claimed phenomenon) actually audible? It's still a fundamental and necessary control for any sort of sensory evaluation for those that do these experiments. Not just audio, any sensory test requires controls. My own introduction to sensory testing was haptic, and double blind was an absolute necessity to get valid data. Ditto when I started doing organoleptic testing and research. Auditory is no different in that respect.

but wouldn't the controls for any perceptual test need to be age matched given that most sensory abilities decline with age?

 

[bboris77]

You may not see the purpose, but the purpose exists nonetheless- is any measured difference (or claimed phenomenon) actually audible? It's still a fundamental and necessary control for any sort of sensory evaluation for those that do these experiments. Not just audio, any sensory test requires controls. My own introduction to sensory testing was haptic, and double blind was an absolute necessity to get valid data. Ditto when I started doing organoleptic testing and research. Auditory is no different in that respect.

Agreed completely that those controls are super important and necessary to have any validity to the data. I never questioned the purpose of the controls to any listening tests. I questioned the usefulness of these listening tests when it's already been established that a device that measures well using the conventional instrumentation-based approach is transparent to the listener.

Ultimately, my worry is that a potential shift towards listening tests, no matter how well-controlled they are, will put the focus more on the performance of the listener and their ability to discern between different equipment rather than say anything about the performance of the equipment itself. And since all well-measuring equipment is transparent, why would we even need to go there?

 

[amirm]

How well did finding audible issues correlate with anomalies in null testing?

Null testing is improper as far as assessing level of audibility. They remove masking effect so create a different situation than listening to the original clip. As a result, we did NOT use null testing for impairments.

 

[Galliardist]

Talk to the marketing futzies.

I would not write that. In fact, had I had my way I would have never had these fuses offered for sale.

Thor

Thanks. As you've confirmed that you're not involved in the company any more, I'll take this claim up directly with AMR.

Pathetic. Really pathetic.

I'll defend my comment. As far as I understand it, AMR was selling audiophile fuses when Thorsten Loesch was listed as being in charge of design, and I believe my claim that he was as "fuse seller" is correct. He's outlined the circumstances in which that happened - clearly there's more he could say, but I don't see the need. The claim that my comment was "pathetic" can only really apply if he was not involved at all and if all audiophile fuse products are sold fraudulently.

For many of these products I wouldn't want to go further than challenging their effectiveness. I do hold that it is for those selling such products for profit, to show evidence for the claims they make for these products, since they should have acquired such during development, shouldn't they? And if marketers make s**t up, presumably senior management and company owners still authorise it to be used to sell the product.

 

[Holmz]

Well, when you claim 40 to 60 dB is the limit of how good a speaker gets, you deserved that response. And I don't need to do any of that when I have had the AP software read the level at 80+ SINAD, which massively disputes your claim.

Was that speakers of headphones?
Which speakers were they?

 

[Holmz]

…
Indeed, we need to do this more often. If you ever come to Thailand, let me know. First shout is mine.

Thor

BKK?
That German beer hall is worth a visit.,, IMO.

 

[amirm]

Was that speakers of headphones?
Which speakers were they?

I don't know about his example. Mine were indicated on the graphs.

 

[birdog1960]

Null testing is improper as far as assessing level of audibility. They remove masking effect so create a different situation than listening to the original clip. As a result, we did NOT use null testing for impairments.

Perhaps I'm understanding your point incorrectly. https://education.arcus.chop.edu/null-hypothesis-testing/ Set me straight if i did. Sensory "impairments" with age are normal. There's only one alternative to aging and it's not good.

 

[SIY]

but wouldn't the controls for any perceptual test need to be age matched given that most sensory abilities decline with age?

Controls? No.

 

[blackmetalboon]

Perhaps I'm understanding your point incorrectly. https://education.arcus.chop.edu/null-hypothesis-testing/ Set me straight if i did. Sensory "impairments" with age are normal. There's only one alternative to aging and it's not good.

No. My experiences with hifi forums seems to show that hearing acuity proportionally corresponds to your age and disposable
income.

 

[birdog1960]

No. My experiences with hifi forums seems to show that hearing acuity proportionally corresponds to your age and disposable
income.

Which is precisely why it's an important variable.

 

[blackmetalboon]

No. My experiences with hifi forums seems to show that hearing acuity proportionally corresponds to your age and expendable income.

Which is precisely why it's an important variable.

Really?