Why haven't subjectivists and objectivists met to do a live ABX test?

[rsoffer]

Please correct me, but I don't believe this has happened. If so, please show me where.

There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:

Magni 3 Heresy and 3+ Blind Listening

Related Threads (background): https://www.superbestaudiofriends.org/index.php?threads/makmeksams-good-measuring-vs-good-sounding-thread.8661/...

www.superbestaudiofriends.org

There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?

In that link above, he claims its very easy to tell the difference between those budget Magni amps. Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?

 

[dshreter]

If you do a blind level matched ABX test and find differences that seems to me that is an objective methodology.

 

[LTig]

Please correct me, but I don't believe this has happened. If so, please show me where.

There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:

Magni 3 Heresy and 3+ Blind Listening

Related Threads (background): https://www.superbestaudiofriends.org/index.php?threads/makmeksams-good-measuring-vs-good-sounding-thread.8661/...

www.superbestaudiofriends.org

There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?

In that link above, he claims its very easy to tell the difference between those budget Magni amps. Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?

It will not help because when the subjectivists fail the ABX test they blame the ABX equipment or test stress or what ever. Been there, done that.

 

[Blumlein 88]

Search for Steve Zipser. Blind test. On the web.

 

[Killingbeans]

There are even more subjectivists who report hearing large differences every day without doing testing.

I'd argue that this actually speaks more for the case of placebo than against it.

In that link above, he claims its very easy to tell the difference between those budget Magni amps.

He does. He claims a lot of things. But he shows zero test results pointing to statistical significance.

Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

Not if the super high end amp is equally well measuring.

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?

Possibly. If it was a well planned and executed test.

 

[JSmith]

DAC blind tests? EVER?

I’ve been searching near and far via google and can’t find any blind abx tests done on DACs with statistically significant results favoring one DAC over another. Does anyone know of any? If so can you please paste a link. If one DAC can significantly improve sound quality of my 2-channel...

www.audiosciencereview.com

JSmith

 

[rsoffer]

I'd argue that this actually speaks more for the case of placebo than against it.



He does. He claims a lot of things. But he shows zero test results pointing to statistical significance.



Not if the super high end amp is equally well measuring.



Possibly. If it was a well planned and executed test.

I think the correct methodology would be that the subject brings his own amps and headphones.

Amir or someone with measurement gear tests the amps to ensure they’re functioning properly and measuring well.

Amir or tester provides ABX switching mechanism.

Set a number of attempts (20?)

Boom. Should honestly take an hour or two?

 

[rsoffer]

DAC blind tests? EVER?

I’ve been searching near and far via google and can’t find any blind abx tests done on DACs with statistically significant results favoring one DAC over another. Does anyone know of any? If so can you please paste a link. If one DAC can significantly improve sound quality of my 2-channel...

www.audiosciencereview.com

JSmith

I know many tests and studies were conducted, but there’s always reported issues.

In this case, the experienced listener subject is bringing their own gear they are familiar with and doing it live. I feel it’s a different scenario.

 

[DonR]

Catalogue of blind tests

I thought this thread on Head-Fi was a valiant effort to put together blind tests that have been performed over the years. Given that there have been many more (I'm thinking Archimago's tests, among others), it would be fun to open source a complete list...

www.audiosciencereview.com

 

[Killingbeans]

I know many tests and studies were conducted, but there’s always reported issues.

In this case, the experienced listener subject is bringing their own gear they are familiar with and doing it live. I feel it’s a different scenario.

It was the same thing with James Randi's One Million Dollar Paranormal Challenge. Every time he exposed a claim as the complete and utter bullshit that it was, it didn't take long for the person who was tested to start "reporting issues". It's just a pill that people refuse to swallow.

 

[rsoffer]

Catalogue of blind tests

I thought this thread on Head-Fi was a valiant effort to put together blind tests that have been performed over the years. Given that there have been many more (I'm thinking Archimago's tests, among others), it would be fun to open source a complete list...

www.audiosciencereview.com

Just to reiterate, what I’m suggesting is different from what I believe all previous tests have been because all that is asked of the subject is to reproduce a test they’ve already claimed they’ve done and succeeded with gear they’re already familiar with.

If you can do it at home with those two exact amps, it should be much easier to reproduce when administered by an Amir.

 

[DonR]

Just to reiterate, what I’m suggesting is different from what I believe all previous tests have been because all that is asked of the subject is to reproduce a test they’ve already claimed they’ve done and succeeded with gear they’re already familiar with.

If you can do it at home with those two exact amps, it should be much easier to reproduce when administered by an Amir.

The Zipser test outlined in that thread was done at the subject's studio on his equipment. Only an unfamiliar amp was swapped out.

 

[rsoffer]

The Zipser test outlined in that thread was done at the subject's studio on his equipment. Only an unfamiliar amp was swapped out.

Yeah, and unfortunately that's what its detractors will point to.

Most of the ABX tests conducted were against subjects not familiar with ABX tests prior, not familiar with the gear involved, etc. There are many experienced subjectivists out there who insist they can hear these differences in blind listening. In their experience, they've already heard these differences in blind listening. Those would be the listeners I'd be interested in testing. Ideally all they would do is reproduce a test they've already conducted themselves.

 

[JSmith]

I feel it’s a different scenario.

Fair enough... however this attempt was made (video US/UK), but didn't proceed;

Blind test: we have a volunteer!!!

In the MQA thread I pointed out the OP had post a video where he made all kinds of observations about the sound of Schiit Magnius and how it was not a good product. Here it is if you want to (or can) watch it: The reviewer responded that he is ready to do such a test so I thought I create a...

www.audiosciencereview.com

JSmith

 

[Chrispy]

My first thought is like why don't libs and conservatives get together and just work things out? LOL. Defining a group is hard enough, let alone finding particularly relevant members of each, let alone getting them together for any type of actual abx testing....my head spins already.

 

[amirm]

There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?

It is incredibly difficult to get subjectivists to agree to any kind of blind tests. I have offered as much as $1,000 and still no takers.

It has happened though. Here is a very high-end audiophile accepting such a test about his cables and losing the challenge: https://www.audiosciencereview.com/...s/audiophile-optimizer.1132/page-5#post-29952

Sadly despite that, he continues to be a hardcore subjectivist putting no value on measurements or blind testing.

 

[amirm]

There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:

Magni 3 Heresy and 3+ Blind Listening

Related Threads (background): https://www.superbestaudiofriends.org/index.php?threads/makmeksams-good-measuring-vs-good-sounding-thread.8661/...

www.superbestaudiofriends.org

Sorry, what kind of example is that? I read through the 3 pages quickly but did not see any proper results reported. No blind test is valid without statistical analysis. Anyone can get a few instances right. The trick is to get > 95% confidence that there was no guess. In these cases, I actually like to see 100% right seeing how people claim night and day differences. To get there, they must run the test at least 10 times. For 95% confidence, they need to get 9 answers right. I routinely run blind tests where I get 5 ot 6 right out of 10 and then get lost after that, showing that those were lucky guesses.

The last pages of that thread are just fiction. See this example:

He has this completely backward. The issue here is Schiit Yggdrasil DAC. I reviewed it in detail and found that it had clear linearity error due to truncation of 24 bit samples to its 20 bit internal resolution (dither should have been used, not truncation). Jude was pulled in to defend the company but didn't know how to run this test. Audio Precision called me to ask how the test is run and I clearly explained a) what it was and b) the filtering used in there. They then conveyed that to Jude and he runs his test. But gets different results. I then get a second sample of the same DAC and show the same problem. This second unit was sent to Schiit but they measured it after updating its firmware, resulting in better measurements than mine (likely they added dither after my findings).

The linearity test and how I run it by the way, originally was written by Audio Precision for their old analyzers. I started running it there. And when I got my new AP, I just recreated it. Folks who had only used the new AP, like Jude, naturally didn't know what it was and how to run it. So the notion that I didn't know how the test works and Jude, etc. did, is just nonsense.

So please don't go by what these people do or say there. Instead, see this video I created on how to properly run blind tests:

 

[Newman]

It is incredibly difficult to get subjectivists to agree to any kind of blind tests. I have offered as much as $1,000 and still no takers.

That seems to fully answer the thread topic/question.

Time to close and move on?

 

[rsoffer]

Sorry, what kind of example is that? I read through the 3 pages quickly but did not see any proper results reported. No blind test is valid without statistical analysis. Anyone can get a few instances right. The trick is to get > 95% confidence that there was no guess. In these cases, I actually like to see 100% right seeing how people claim night and day differences. To get there, they must run the test at least 10 times. For 95% confidence, they need to get 9 answers right. I routinely run blind tests where I get 5 ot 6 right out of 10 and then get lost after that, showing that those were lucky guesses.

The last pages of that thread are just fiction. See this example:
View attachment 250371

He has this completely backward. The issue here is Schiit Yggdrasil DAC. I reviewed it in detail and found that it had clear linearity error due to truncation of 24 bit samples to its 20 bit internal resolution (dither should have been used, not truncation). Jude was pulled in to defend the company but didn't know how to run this test. Audio Precision called me to ask how the test is run and I clearly explained a) what it was and b) the filtering used in there. They then conveyed that to Jude and he runs his test. But gets different results. I then get a second sample of the same DAC and show the same problem. This second unit was sent to Schiit but they measured it after updating its firmware, resulting in better measurements than mine (likely they added dither after my findings).

The linearity test and how I run it by the way, originally was written by Audio Precision for their old analyzers. I started running it there. And when I got my new AP, I just recreated it. Folks who had only used the new AP, like Jude, naturally didn't know what it was and how to run it. So the notion that I didn't know how the test works and Jude, etc. did, is just nonsense.

So please don't go by what these people do or say there. Instead, see this video I created on how to properly run blind tests:

Sorry, didn't mean this was a good example of an A/B test or meaningful. Just that they *believe* its meaningful. Also didn't realize they were taking digs at you there.

 

[rsoffer]

Fair enough... however this attempt was made (video US/UK), but didn't proceed;

Blind test: we have a volunteer!!!

In the MQA thread I pointed out the OP had post a video where he made all kinds of observations about the sound of Schiit Magnius and how it was not a good product. Here it is if you want to (or can) watch it: The reviewer responded that he is ready to do such a test so I thought I create a...

www.audiosciencereview.com

JSmith

It is incredibly difficult to get subjectivists to agree to any kind of blind tests. I have offered as much as $1,000 and still no takers.

It has happened though. Here is a very high-end audiophile accepting such a test about his cables and losing the challenge: https://www.audiosciencereview.com/...s/audiophile-optimizer.1132/page-5#post-29952

Sadly despite that, he continues to be a hardcore subjectivist putting no value on measurements or blind testing.

Yeah, the whole saga with GoldenSound was pretty rough and sad to see. Lots of bad blood there, and it was hyped to a level where the stakes became too high too early. Money involved, reputations on the line, this is not how a test should be conducted.

Ideally, this would be a very laid back and relaxed environment without all this ill will. The test should be conducted with the understanding that if they PASS the initial test, a more rigorous one would probably follow that is more scrutinized. It hard enough to have any test conducted at all. Also, these should happen in person after the tester and subject have had a chance to get to know each other.

It really seemed like GoldenSound had the right idea about how the test should be conducted, but politics got involved and amir got swept up in the personal attacks which came prior.