Master Thread: Are measurements Everything or Nothing?

[voodooless]

How do you explain this?

Please not this again Do proper testing with a statistically significant number of DAC’s and then we’ll see.

 

[manisandher]

Please not this again

Relax. No need to get your knickers in a twist.

Do proper testing with a statistically significant number of DAC’s and then we’ll see.

I asked the developer of DeltaWave a question. He answered. Nuff said.

 

[Mr. Widget]

I can't pretend to be as knowledgeable in the fundamentals as several members here, my math skills are poor, and I never studied statistics, but I too see the logic in ABX testing. That said, I also have an open mind to the notion that we might still be missing something in this debate.

Years ago I had a preamp that allowed switching absolute phase from the remote. The sound was different, but I could never hear one phase or the opposite as "correct". I wonder if instantaneous changes are the end all as believed.

Perhaps a better test would be a longer term ABX? One where over several weeks you are able switch between components (not knowing what the changes are) presumably you would make changes choosing the one you prefer. After a lengthy comparison the hours of play for the various options would be tallied by the automated test setup. Just throwing this out there as some in the subjectivist camp maintain longer listening is required to determine accurate preference.

I fully agree that sighted bias is real. However the idea that all subjective perceptions are wrong if they have not been backed by a controlled ABX test is narrow minded and comes with its own confirmation bias.

 

[SIY]

I can't pretend to be as knowledgeable in the fundamentals as several members here, my math skills are poor, and I never studied statistics, but I too see the logic in ABX testing. That said, I also have an open mind to the notion that we might still be missing something in this debate.

Years ago I had a preamp that allowed switching absolute phase from the remote. The sound was different, but I could never hear one phase or the opposite as "correct". I wonder if instantaneous changes are the end all as believed.

Perhaps a better test would be a longer term ABX? One where over several weeks you are able switch between components (not knowing what the changes are) presumably you would make changes choosing the one you prefer. After a lengthy comparison the hours of play for the various options would be tallied by the automated test setup. Just throwing this out there as some in the subjectivist camp maintain longer listening is required to determine accurate preference.

I fully agree that sighted bias is real. However the idea that all subjective perceptions are wrong if they have not been backed by a controlled ABX test is narrow minded and comes with its own confirmation bias.

First, it is EXTREMELY important not to conflate "ABX" with "use of basic controls like double blind and level match." ABX is a specific test format and there's lots of alternatives.

There is no alternative to using basic controls. And if an extraordinary claim is made without those basic controls done, it is frankly worthless. That may seem harsh, but it is nonetheless true.

 

[Reynaldo]

I don’t know who he is at all. I don’t actually care. What I do know is that he is using a tactic common to hucksters and has flat out said he is baiting (red flag waiving) and intentionally not posting evidence. So at best he is not arguing in good faith. But he is, to me, indistinguishable from a troll and using the techniques frauds use, apparently deliberately. It is not a good look.

When Chinese DAC makers release a product every few months to get a higher number on SINAD, is it good faith practice?
Many users associated this with better quality devices.

 

[Mr. Widget]

First, it is EXTREMELY important not to conflate "ABX" with "use of basic controls like double blind and level match." ABX is a specific test format and there's lots of alternatives.

There is no alternative to using basic controls. And if an extraordinary claim is made without those basic controls done, it is frankly worthless. That may seem harsh, but it is nonetheless true.

No argument on the first part, I was using ABX as shorthand...

However to your second point, "worthless" is overstating it. It is always possible that uncontrolled results can lead to valid questions and to deeper exploration to test the validity of the uncontrolled findings.

 

[Purité Audio]

Without controls then it is just anecdote,
Keith

 

[Mr. Widget]

Without controls then it is just anecdote,
Keith

Yes, or looked at another way, it can be a question that should be answered.

 

[Purité Audio]

First you must establish if there is a question, if there is a difference.
Keith

 

[Mr. Widget]

First you must establish if there is a question, if there is a difference.
Keith

Agreed.

I am not convinced that we have established that there is no audible difference. I am also not certain that an audible difference exists.

I am not trolling for more debate. I think that is a silly waste of time, but I would like to see more study on the audible thresholds regarding digital audio. I am aware of several conducted in the search for lower resolution compression algorithms, but not for more mainstream higher resolution audio devices.

 

[Geert]

I am not convinced that we have established that there is no audible difference.

We didn't. We've only been waiting for more than 30 years for those who do to come with a well substantiated case that requires more research. Doesn't need to be ABX, as longs as the test methodology is sound.

 

[IPunchCholla]

When Chinese DAC makers release a product every few months to get a higher number on SINAD, is it good faith practice?
Many users associated this with better quality devices.

Whataboutism much? I said nothing about DACs. Chinese or otherwise. Why are you assuming my stance on them or SINAD?

 

[Newman]

It is always possible that uncontrolled results can lead to valid questions and to deeper exploration to test the validity of the uncontrolled findings.

“Possible”, yes, but that doesn’t mean it’s a fruitful way to determine what to put to “deeper exploration” ie controlled testing.

Especially since we are talking about “uncontrolled results” as sighted listening. Too much dominated by non-sonic factors.

That’s why the serious audio research community doesn’t waste time/money on chasing down silly audiophile myths that arose from uncontrolled results. Even though that’s exactly what the audiophile community wants them to do.

 

[Mr. Widget]

We've only been waiting for more than 30 years for those who do to come with a well substantiated case that requires more research.

Yeah, I'm still waiting along with you.

“Possible”, yes, but ...

I guess the good news is that audio playback equipment has gotten so good that we find ourselves discussing and at times arguing about rounding errors.

 

[amirm]

Amir, you are being patronising and seem to think I'm simpleton.

"look how close it hugs the 0% line".

Please use a logarithmic scale and show me how close it hugs the 0.1% Line (60dB SINAD), 0.01% line (80dB SINAD) and 0.001% line (100dB SINAD).

Well, when you claim 40 to 60 dB is the limit of how good a speaker gets, you deserved that response. And I don't need to do any of that when I have had the AP software read the level at 80+ SINAD, which massively disputes your claim.

 

[Reynaldo]

Whataboutism much? I said nothing about DACs. Chinese or otherwise. Why are you assuming my stance on them or SINAD?

I gave an example.
I referred to sales tactics, did you understand?

 

[krabapple]

yes, because cars are way more complicated. But we compare car audio systems. narrow horizons are dumb. I'm looking at it as a consumer (and I'll bet many manufacturers are too). You seem to be looking at it as a pure science, which most would agree, it isn't, in terms of listening enjoyment..If it was, there would be one speaker design, one amp design, one digital or analog source (even with cost as no object) etc that was considered superior in a given room.

Marketing is a hell of a thing.

 

[krabapple]

I am a bad actor because I don't act. I have been on stage, as MC, as DJ, giving talks etc. But I do not act.

Funny you dismiss Oohashi as crank. I guess anything that misaligns with your preconceptions (misconceptions) is just bunkum for you.

I wish I was so certain of myself, with such clear steelyeyed vision into infinity and not the least self doubt.

Thor

Au contraire, whoever you are, you seem quite certain of a number of things.

And at this point, as concerns you, so am I.

I'm also sure I've read more of Oohashi's body of work in science -- he didn't confine his 'genius' to just audio research, btw -- than you have. And judged it accordingly.

 

[IPunchCholla]

I gave an example.
I referred to sales tactics, did you understand?

And what does gaming SINAD have to do with what I was addressing? You seem to be supposing I am somehow for it?

Regardless, the end conclusion of you positing it is that if I am against one thing, I should seek out and point out every example of that thing, and since I didn’t do that, I shouldn’t have pointed out the flimflammery I did point out.

Or something.

 

[krabapple]

Wellll I dunno.
Forced choice test... yes. That's rather a property of a test which is to be used what it is designed for.

Can you explain why you think statistics are an issue. The more attempts the clearer the 'view' but the easier it becomes to make an honest mistake due to time factors.

He's banging on about Type I vs II errors. This has been going on since at least the mid 1980s when Les Leventhal banged on about it in Stereophile.

Both LL and this new actor neglect the context of the typical audiophile claim, going back to the 1980s: "I changed something and immediately heard things get better'

If you want to rule out *tout court* to our highest standard that A and B do not sound different, you'll want to design sufficient statistical power into your research plan to minimize type II errors. Lots of trials, essentially (assuming the rest of your method is ship shape re: randomizing, level matching, listener training, choice of samples, etc)

THAT IS *SCIENTIFIC RESEARCH FOR PUBLICATION*. It is testing the limits of human hearing.

That is isn't AT ALL what is required to debunk a particular audiophile's claims about what he heard -- or Mr . Torsten's own person claims of what he hears.

In the typical golden ear case, the golden ear already claims to hear A vs B quite well, thank you. Sighted.
The simplest test of that is to blind the golden ear, level match his samples, and let that golden ear try to do it just as he did before. (An ABX can be an efficient way to do taht, though it's not the only double blind listening protocol). Can he still do it when the very simplest controls are put in place?

You don't need an elephant gun to kill a mouse. If the golden ear is willing, it is very easy to determine with high confidence whether *he* heard what *he* claims to have heard. Vast numbers of trials are NOT required. A proctored test is.

Mr Torsten is also banging on about the 'ABX company' , as if there was something sinister about David Clark designing and selling a switching box (the ABX Comparator) that made it possible for hobbyists to perform level matched hardware ABX comparisons. Do it yourself designs have since long been published online (and Clark's company, AFAIK, went out business decades ago). Meanwhile, of course, software ABX has revolutionized and democratized the use of blind testing to compare audio files for audible difference.