• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Why haven't subjectivists and objectivists met to do a live ABX test?

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76
Please correct me, but I don't believe this has happened. If so, please show me where.

There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:

There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?

In that link above, he claims its very easy to tell the difference between those budget Magni amps. Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?
 

LTig

Master Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
5,760
Likes
9,442
Location
Europe
Please correct me, but I don't believe this has happened. If so, please show me where.

There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:

There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?

In that link above, he claims its very easy to tell the difference between those budget Magni amps. Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?
It will not help because when the subjectivists fail the ABX test they blame the ABX equipment or test stress or what ever. Been there, done that.
 

Killingbeans

Major Contributor
Joined
Oct 23, 2018
Messages
4,088
Likes
7,544
Location
Bjerringbro, Denmark.
There are even more subjectivists who report hearing large differences every day without doing testing.

I'd argue that this actually speaks more for the case of placebo than against it.

In that link above, he claims its very easy to tell the difference between those budget Magni amps.

He does. He claims a lot of things. But he shows zero test results pointing to statistical significance.

Wouldn't it be even easier to tell with a super high end amp vs. a budget well measuring amp?

Not if the super high end amp is equally well measuring.

If this live test took place, and one subjectivist was able to reliably tell the difference between these well measuring amps. Would that be enough to change your opinion on the matter?

Possibly. If it was a well planned and executed test.
 

JSmith

Master Contributor
Joined
Feb 8, 2021
Messages
5,152
Likes
13,211
Location
Algol Perseus


JSmith
 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76
I'd argue that this actually speaks more for the case of placebo than against it.



He does. He claims a lot of things. But he shows zero test results pointing to statistical significance.



Not if the super high end amp is equally well measuring.



Possibly. If it was a well planned and executed test.

I think the correct methodology would be that the subject brings his own amps and headphones.

Amir or someone with measurement gear tests the amps to ensure they’re functioning properly and measuring well.

Amir or tester provides ABX switching mechanism.

Set a number of attempts (20?)

Boom. Should honestly take an hour or two?
 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76


JSmith

I know many tests and studies were conducted, but there’s always reported issues.

In this case, the experienced listener subject is bringing their own gear they are familiar with and doing it live. I feel it’s a different scenario.
 

DonR

Major Contributor
Joined
Jan 25, 2022
Messages
2,968
Likes
5,611
Location
Vancouver(ish)
 

Killingbeans

Major Contributor
Joined
Oct 23, 2018
Messages
4,088
Likes
7,544
Location
Bjerringbro, Denmark.
I know many tests and studies were conducted, but there’s always reported issues.

In this case, the experienced listener subject is bringing their own gear they are familiar with and doing it live. I feel it’s a different scenario.

It was the same thing with James Randi's One Million Dollar Paranormal Challenge. Every time he exposed a claim as the complete and utter bullshit that it was, it didn't take long for the person who was tested to start "reporting issues". It's just a pill that people refuse to swallow.
 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76

Just to reiterate, what I’m suggesting is different from what I believe all previous tests have been because all that is asked of the subject is to reproduce a test they’ve already claimed they’ve done and succeeded with gear they’re already familiar with.

If you can do it at home with those two exact amps, it should be much easier to reproduce when administered by an Amir.
 

DonR

Major Contributor
Joined
Jan 25, 2022
Messages
2,968
Likes
5,611
Location
Vancouver(ish)
Just to reiterate, what I’m suggesting is different from what I believe all previous tests have been because all that is asked of the subject is to reproduce a test they’ve already claimed they’ve done and succeeded with gear they’re already familiar with.

If you can do it at home with those two exact amps, it should be much easier to reproduce when administered by an Amir.
The Zipser test outlined in that thread was done at the subject's studio on his equipment. Only an unfamiliar amp was swapped out.
 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76
The Zipser test outlined in that thread was done at the subject's studio on his equipment. Only an unfamiliar amp was swapped out.
Yeah, and unfortunately that's what its detractors will point to.

Most of the ABX tests conducted were against subjects not familiar with ABX tests prior, not familiar with the gear involved, etc. There are many experienced subjectivists out there who insist they can hear these differences in blind listening. In their experience, they've already heard these differences in blind listening. Those would be the listeners I'd be interested in testing. Ideally all they would do is reproduce a test they've already conducted themselves.
 

JSmith

Master Contributor
Joined
Feb 8, 2021
Messages
5,152
Likes
13,211
Location
Algol Perseus
I feel it’s a different scenario.
Fair enough... however this attempt was made (video US/UK), but didn't proceed;


JSmith
 

Chrispy

Master Contributor
Forum Donor
Joined
Feb 7, 2020
Messages
7,751
Likes
5,910
Location
PNW
My first thought is like why don't libs and conservatives get together and just work things out? LOL. Defining a group is hard enough, let alone finding particularly relevant members of each, let alone getting them together for any type of actual abx testing....my head spins already.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
There are even more subjectivists who report hearing large differences every day without doing testing. The objectivist claim is that well measuring gear is audibly indistinguishable. Wouldn't it be super easy to debunk with one live ABX test using gear the subjective listener is familiar with?
It is incredibly difficult to get subjectivists to agree to any kind of blind tests. I have offered as much as $1,000 and still no takers.

It has happened though. Here is a very high-end audiophile accepting such a test about his cables and losing the challenge: https://www.audiosciencereview.com/...s/audiophile-optimizer.1132/page-5#post-29952

Sadly despite that, he continues to be a hardcore subjectivist putting no value on measurements or blind testing.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
There's many subjectivists out there who have done ABX testing and report hearing differences. Here is an example:
Sorry, what kind of example is that? I read through the 3 pages quickly but did not see any proper results reported. No blind test is valid without statistical analysis. Anyone can get a few instances right. The trick is to get > 95% confidence that there was no guess. In these cases, I actually like to see 100% right seeing how people claim night and day differences. To get there, they must run the test at least 10 times. For 95% confidence, they need to get 9 answers right. I routinely run blind tests where I get 5 ot 6 right out of 10 and then get lost after that, showing that those were lucky guesses.

The last pages of that thread are just fiction. See this example:
1671159641118.png


He has this completely backward. The issue here is Schiit Yggdrasil DAC. I reviewed it in detail and found that it had clear linearity error due to truncation of 24 bit samples to its 20 bit internal resolution (dither should have been used, not truncation). Jude was pulled in to defend the company but didn't know how to run this test. Audio Precision called me to ask how the test is run and I clearly explained a) what it was and b) the filtering used in there. They then conveyed that to Jude and he runs his test. But gets different results. I then get a second sample of the same DAC and show the same problem. This second unit was sent to Schiit but they measured it after updating its firmware, resulting in better measurements than mine (likely they added dither after my findings).

The linearity test and how I run it by the way, originally was written by Audio Precision for their old analyzers. I started running it there. And when I got my new AP, I just recreated it. Folks who had only used the new AP, like Jude, naturally didn't know what it was and how to run it. So the notion that I didn't know how the test works and Jude, etc. did, is just nonsense.

So please don't go by what these people do or say there. Instead, see this video I created on how to properly run blind tests:

 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76
Sorry, what kind of example is that? I read through the 3 pages quickly but did not see any proper results reported. No blind test is valid without statistical analysis. Anyone can get a few instances right. The trick is to get > 95% confidence that there was no guess. In these cases, I actually like to see 100% right seeing how people claim night and day differences. To get there, they must run the test at least 10 times. For 95% confidence, they need to get 9 answers right. I routinely run blind tests where I get 5 ot 6 right out of 10 and then get lost after that, showing that those were lucky guesses.

The last pages of that thread are just fiction. See this example:
View attachment 250371

He has this completely backward. The issue here is Schiit Yggdrasil DAC. I reviewed it in detail and found that it had clear linearity error due to truncation of 24 bit samples to its 20 bit internal resolution (dither should have been used, not truncation). Jude was pulled in to defend the company but didn't know how to run this test. Audio Precision called me to ask how the test is run and I clearly explained a) what it was and b) the filtering used in there. They then conveyed that to Jude and he runs his test. But gets different results. I then get a second sample of the same DAC and show the same problem. This second unit was sent to Schiit but they measured it after updating its firmware, resulting in better measurements than mine (likely they added dither after my findings).

The linearity test and how I run it by the way, originally was written by Audio Precision for their old analyzers. I started running it there. And when I got my new AP, I just recreated it. Folks who had only used the new AP, like Jude, naturally didn't know what it was and how to run it. So the notion that I didn't know how the test works and Jude, etc. did, is just nonsense.

So please don't go by what these people do or say there. Instead, see this video I created on how to properly run blind tests:


Sorry, didn't mean this was a good example of an A/B test or meaningful. Just that they *believe* its meaningful. Also didn't realize they were taking digs at you there.
 
OP
R

rsoffer

Member
Joined
Sep 29, 2022
Messages
69
Likes
76
Fair enough... however this attempt was made (video US/UK), but didn't proceed;


JSmith

It is incredibly difficult to get subjectivists to agree to any kind of blind tests. I have offered as much as $1,000 and still no takers.

It has happened though. Here is a very high-end audiophile accepting such a test about his cables and losing the challenge: https://www.audiosciencereview.com/...s/audiophile-optimizer.1132/page-5#post-29952

Sadly despite that, he continues to be a hardcore subjectivist putting no value on measurements or blind testing.

Yeah, the whole saga with GoldenSound was pretty rough and sad to see. Lots of bad blood there, and it was hyped to a level where the stakes became too high too early. Money involved, reputations on the line, this is not how a test should be conducted.

Ideally, this would be a very laid back and relaxed environment without all this ill will. The test should be conducted with the understanding that if they PASS the initial test, a more rigorous one would probably follow that is more scrutinized. It hard enough to have any test conducted at all. Also, these should happen in person after the tester and subject have had a chance to get to know each other.

It really seemed like GoldenSound had the right idea about how the test should be conducted, but politics got involved and amir got swept up in the personal attacks which came prior.
 
Top Bottom