Are DBTs the defacto standard for Audio Science research?

[March Audio]

." The "high-end" (I find it almost impossible to type without qualifying quotation marks) world is teeming with bias and self-delusion, and the "marketing" of "high-end" is full of fraud. I approach all "audiophile" products, especially the absurdly expensive ones, with a high level of skepticism, and the ears of "high-end" hobbyists are the last thing I'd trust.



Tim

My position also

 

[SoundAndMotion]

Whilst there may be good reasons to reject a casually performed blind test, it can of course be flawed, it doesnt automatically follow that they are flawed or invalid.

I agree and that's what I tell John, but his answer is "how does one know which are good". I answer from the description of methods, but they are often insufficient. John and I agree to disagree.

However there is no good reason to accept any findings of a sighted test. The subject will be influenced by various biases.

Perfection is the enemy of progress. Of the 1000's of scientific papers I've read, NONE is perfect. There is always room for improvement, which means it was not perfect. It is a matter of carefully assessing the severity of the flaws. There are papers where the flaws prevent me accepting the conclusions of the authors, but without knowing the hypothesis to be tested and the exact methods used, I cannot a priori reject outright a sighted test. I agree that if all other things are equal and being sighted is not required for the test, a blind test is less flawed than a sighted one.

The problem also exists where people choose the evidence that supports their position, which I think is what has been legitimately highlighted here and in another thread.

Hmmm, don't we all do that? Don't you do that for evidence that supports a view with which you disagree? In a paper's discussion, one should highlight potential weaknesses or known flaws, but I don't believe I've ever seen that in forum posts.

 

[March Audio]

I agree and that's what I tell John, but his answer is "how does one know which are good". I answer from the description of methods, but they are often insufficient. John and I agree to disagree.


Perfection is the enemy of progress. Of the 1000's of scientific papers I've read and certainly the dozens I've written, NONE is perfect. It is a matter of carefully assessing the severity of the flaws. There are papers where the flaws prevent me accepting the conclusions of the authors, but without knowing the hypothesis to be tested and the exact methods used, I cannot a priori reject outright a sighted test. I agree that if all other things are equal and being sighted is not required for the test, a blind test is less flawed than a sighted one.

Hmmm, don't we all do that? Don't you do that for evidence that supports a view with which you disagree? In a paper's discussion, one should highlight potential weaknesses or known flaws, but I don't believe I've ever seen that in forum posts.

Agreed.

I dont think its about perfection in this instance. Any sighted audio comparison will incur bias of one nature or another. People like to think they are immune, but they are not. Thats enough to reject outright. Its not objective, its simply "I like".

Well, yes we probably are all guilty of that to a degree at some time or another, however some people seek the truth and can be objective, some people just seek affirmation.

 

[SoundAndMotion]

I dont think its about perfection in this instance. Any sighted audio comparison will incur bias of one nature or another. People like to think they are immune, but they are not.

I agree!

Thats enough to reject outright.

I disagree! Without knowing the purpose, or other aspects of the methods, I can't say "outright". A simple example, with which I'd guess you agree: put 2 identical devices in 2 different enclosures to see whether one enclosure produces a stronger bias. It is a "sighted audio comparison", but the sight is necessary. I know you had a different intent with "sighted audio comparison", but I just can't say "outright" without the exact intent being stated. That is not intended to be pedantic, but rather a necessary obsession with details, as in papers we write or read.

 

[Phelonious Ponk]

The perpetual audiophile argument against blind listening is really just an attempt to legitimize subjectivism by appearing to discrediting science. The "flaws" that are often used in these arguments are laughable compared to the overwhelming dysfunction of sighted listening. It doesn't even deserve to be connected to the word "test." If a blind test only succeeds in preventing the listener from knowing what components he is comparing - what they look like, what brand and reputation they carry, it has eliminated the greatest problem with sighted listening. A flawed, casual, blind test is miles ahead of a perfect sighted one, because no amount of statistical significance and scientific rigor can overcome the expectation bias created by sighted listening comparisons.

Tim

 

[March Audio]

I agree!

I disagree! Without knowing the purpose, or other aspects of the methods, I can't say "outright". A simple example, with which I'd guess you agree: put 2 identical devices in 2 different enclosures to see whether one enclosure produces a stronger bias. It is a "sighted audio comparison", but the sight is necessary. I know you had a different intent with "sighted audio comparison", but I just can't say "outright" without the exact intent being stated. That is not intended to be pedantic, but rather a necessary obsession with details, as in papers we write or read.

If you understood the context, as you allude to, then it is somewhat pedantic, but Ill let you off cos you did it nicely

 

[March Audio]

The perpetual audiophile argument against blind listening is really just an attempt to legitimize subjectivism by appearing to discrediting science. The "flaws" that are often used in these arguments are laughable compared to the overwhelming dysfunction of sighted listening. It doesn't even deserve to be connected to the word "test." If a blind test only succeeds in preventing the listener from knowing what components he is comparing - what they look like, what brand and reputation they carry, it has eliminated the greatest problem with sighted listening. A flawed, casual, blind test is miles ahead of a perfect sighted one, because no amount of statistical significance and scientific rigor can overcome the expectation bias created by sighted listening comparisons.

Tim

Ditto.

 

[AJ Soundfield]

The perpetual audiophile argument against blind listening is really just an attempt to legitimize subjectivism by appearing to discrediting science.

Precisely.
The whole "it must be 100% perfect science" for the "true champions of science" is a complete facade for wanting their delusions of grandeur to be true, until proven false by "perfect" science.
That's it, period.
It is utter hypocrisy for jkeny et al to posit any form of bias control testing as proof when they utterly reject them.
Stereophile has now openly taken the official position that bias controlled tests are useless for audio...but I've seen the instance where JA and Fremer took a "blind" test at an AES event waved like a trophy dozens of times by the very same people who reject blind tests, as "proof" they can hear this and that...and how great their hearing is vs deaf "engineers".
I need not repeat what condition causes folks to be non-cognizant of such hypocrisy.
I have yet to see an audio dichotomy not explained by D-K.

 

[John Kenny]

AJ, from your OP and this post it is clear you created this thread as an anti-JohnKenny thread. Strange for an audio science forum. I enjoy when your posts contain interesting, scientific content, including references, but isn’t this just a personal vendetta? Is that appropriate?

To be clear, the post from John that you often cite here and in other forums, seems to relate to a specific comparison, where he rejects the need for a casually performed DBT versus subjective listening, and it is nearly 6 years old. I’ve only read posts from John the last year and a half or so, where he clearly states that he accepts professionally-done DBTs, but not those that are casually-done. Because of the uncertainty of such tests, he prefers to reject all that are casually performed. I have argued that point with him many times… quite civilly… and now we agree to disagree (since we both have better things to do than continue to try to persuade).

Rather than focus on one person (or a small group) that you consider the enemy, argue against their posts, not them personally. If you have to attack the person, not their current post, it weakens what may be important, useful points from you.

John and I disagree about as much as we agree, but we both like the science the other brings up (have you seen the threads he started in Psychoacoustics?). Since we both like to discuss science, he is not my enemy, even though we sometimes disagree.

As this is a thinly disguised anti-JK thread (about my views on ABX testing), I don't understand AJ's need to trawl through other forums & link to 6 year old posts when my views on ABX are right here, in ASR, under his very nose:
"My position is that the ABX test has no quality checks within it & therefore requires proctoring by someone who knows how to do perceptual testing. So, home run ABX tests to me are always of questionable quality & I've no real way of knowing what this quality is. And even, on the odd occasion, someone registers a positive ABX test, it doesn't tell me which is better.

As a result I favour anecdotal reports of sighted listening which give equipment used/vol matched tracks used & a description of better/worse differences heard. I can use this information to check the same tracks on my own system for the same factors perceived."

 

[AJ Soundfield]

I don't understand AJ's need to trawl through other forums & link to 6 year old posts when my views on ABX are right here, in ASR, under his very nose:
"My position is that the ABX test has no quality checks within it & therefore requires proctoring by someone who knows how to do perceptual testing. So, home run ABX tests to me are always of questionable quality & I've no real way of knowing what this quality is. And even, on the odd occasion, someone registers a positive ABX test, it doesn't tell me which is better.

As a result I favour anecdotal reports of sighted listening which give equipment used/vol matched tracks used & a description of better/worse differences heard. I can use this information to check the same tracks on my own system for the same factors perceived."

That is exactly what you said (quoted from 6yrs ago) and now revised to exactly I said:
Dismissal of bias controlled testing (DBTs or dog whistle "ABX") under the facade of scientific purity, in favor of anecdotal daydreams expressed as physical reality in the soundfield, not purely psychogenic.
Are you now embarrassed by/distance yourself from those 6yr old words, or do you own up to them as your true beliefs, minus the new facade?

 

[SoundAndMotion]

"SAM" (or whatever "name" you prefer to hide behind), it is equally clear to me your reading skills haven't improved since HA.

My reading skills are great and I won't engage in mud wrestling with you in this forum.

"JKeny" position is consistent with audiophile belief. They, as an official position, reject any form of bias control testing, such as DBTs, which they often dog-whistle conflate with "ABX".
They, as jkeny has stated here openly (please desist with the laziness you displayed at HA and search out his posts where he said so vs me having to do your legwork) that he rejects all controlled testing in favor of anecdotal delusional view-"listening".
**However**, if there is ever an instance where a "blind" test, be it using "tea towels" on PFM, an ad hoc audio show "ABX" (despite no X), or some audio journal, appears to have positive results, then he/audiophiles will accept those results as 100% valid and irrefutable proof of an audiophile belief.
Any negative from the thousands of AES DBTs, to amateur AB or ABX, will be dismissed, due to some fault, primarily, it conflicts with audiophile belief, but guised under the facade of wanting 100% reliable science.
It is of zero concern to me whether you can comprehend the well known issue of audiophile disorder, as described above. You have made it clear you are not involved in any way with audiophiles.

I have not been lazy here or at HA. It is not my legwork, but yours. Your accusation means that when challenged you back it up. For someone who likes to demand evidence, you are shy in providing it.
Sorry AJ, I'm not dressed for playing in the mud.

 

[Phelonious Ponk]

That is exactly what you said (quoted from 6yrs ago) and now revised to exactly I said:
Dismissal of bias controlled testing (DBTs or dog whistle "ABX") under the facade of scientific purity, in favor of anecdotal daydreams expressed as physical reality in the soundfield, not purely psychogenic.
Are you now embarrassed by/distance yourself from those 6yr old words, or do you own up to them as your true beliefs, minus the new facade?

Perhaps he has just changed his mind, AJ. I revised my view of audio, dramatically, about 10 years ago, and continue to revise it incrementally, daily, to this day.

Oh, and back on topic, no, they're not the defacto standard.

Tim

 

[AJ Soundfield]

Perhaps he has just changed his mind, AJ.

No. He, like many audiophiles, realized sensible people don't accept such childish arguments, so he has created the new facade position of "science puritan" guise, while fully maintaining his belief. This isn't about JK.
He is not alone and it is that position I find incompatible with Audio Science.
Can audiophile believers who (in reality) dismiss audio science, allow an "Audio Science" forum to function?
Check out AA's "Propeller Head Plaza" for a sneak preview....

 

[John Kenny]

Then why are you constantly linking "Vital on PFM tea-towel tests" and Joe blow at Audio show T & A preamp "ABX" tests for "soundstage depth" "proof"??

Like SAM, I don't think the mud is the place where intelligent debates can be conducted

 

[Phelonious Ponk]

Like SAM, I don't think the mud is the place where intelligent debates can be conducted

Desisting the back and forth can be really hard, huh?

Tim

 

[Mivera]

You assumed incorrectly. I'm actually in the camp you are discussing with Amir and a couple of others right now. I believe audio electronics are such a mature category that the differences between electronics designed, engineered and built with the objective of achieving the best possible fidelity to the source are insignificant. So I put my faith in companies and designers who believe in no audiophile voodoo, seek no "tonality" or house sound, and whose goals are focused on the accurate reproduction of the recording. It frees me to concentrate on transducers, recordings, listening. The most subjective stuff. I'm not so much a self-declared objectivist as anti-"high-end." The "high-end" (I find it almost impossible to type without qualifying quotation marks) world is teeming with bias and self-delusion, and the "marketing" of "high-end" is full of fraud. I approach all "audiophile" products, especially the absurdly expensive ones, with a high level of skepticism, and the ears of "high-end" hobbyists are the last thing I'd trust.

I suspect that was less than clear, so let me give you an example. In my interest in a DAC/pre, I would simply buy a Benchmark, based on their objectives, philosophy, reputation and yes, the stunning numbers to be found in independent reviews by professional, not hobbyist, publications. If there is an audible difference between the Benchmark and a Vivaldi stack, I suspect that thorough analysis would reveal it is the fault of the Vivaldi, deliberately created by the desire to appeal to audiophiles. But Benchmarks products are deliberately over-engineered and necessarily over-priced, and I believe a product could be made that would be indistinguishable from the Benchmark, even with the full feature set of their DAC/pre, at a fraction of its price. I love a bargain.

And yes, I know we disagree.

Tim

So in other words you look for a DAC manufacturer who can provide you with confidence inspiring marketing material? I haven't heard anything from you about what objective data you look for specifically in a DAC.

 

[AJ Soundfield]

I haven't heard anything from you about what objective data you look for specifically in a DAC.

Nor reliable listening test evidence for yours, specifically, on an Audio Science forum.
Hmmm

 

[Mivera]

Nor reliable listening test evidence for yours, specifically, on an Audio Science forum.
Hmmm

I'm comfortable with my method of choosing components. I was asking Tim what his criteria was. From what I heard, sounds like he puts 100% confidence in the manufacturers marketing material, and this is the main criteria he looks for. Words that meet his standards.

 

[AJ Soundfield]

I'm comfortable with my method of choosing components.

Right, but for someone else, like Tim, would not care how you choose, but rather whether that audio component choice resulted in any audio benefit, vs say a purely psychogenic one.
Where is your audio science/reliable audio evidence in favor of any DAC you would recommend?

 

[Phelonious Ponk]

I'm comfortable with my method of choosing components. I was asking Tim what his criteria was. From what I heard, sounds like he puts 100% confidence in the manufacturers marketing material, and this is the main criteria he looks for. Words that meet his standards.

Unless you consider a lot of reading about design and testing in general, plus the measurement, review and analysis of specific products in pro audio and audiophile skeptic publications, the views of people I trust in places like this, and a deep dive into the research, design and testing methodologies of companies that draw my interest to be the "manufacturers marketing material," that's 100% inaccurate, presumptuous, and shallow. I'm trying very hard to be good, Mike, but keep poking at me this way and I will eventually slap back. It will hurt. What I lack in technical expertise I make up for in command of language.

Tim