• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Blind test - objectivists with tin hearing?

garbulky

Major Contributor
Joined
Feb 14, 2018
Messages
1,510
Likes
827
You don't have to level match. Or follow any protocol other than doing the testing blind. Have someone switch out one product for the other a dozen times once a day and you both keep score of which unit is what. Then at the 12th day compare notes and see how it matches your sighted evaluation.
You may have something there....
I may try such a thing. I could use my harmony universal remote so that I wouldn't know which preamp I was using. I'd just need another DAC, perhaps one I find not as pleasing/different like the Oppo 205.Well actually since my DC-1 is an analog preamp, I could just that as the input for a cheap dac and see if I can hear a difference. I have a $30 Behringer 202 which I thought sounded quite good.
 

Soniclife

Major Contributor
Forum Donor
Joined
Apr 13, 2017
Messages
4,499
Likes
5,417
Location
UK
You may have something there....
I may try such a thing. I could use my harmony universal remote so that I wouldn't know which preamp I was using. I'd just need another DAC, perhaps one I find not as pleasing/different like the Oppo 205.Well actually since my DC-1 is an analog preamp, I could just that as the input for a cheap dac and see if I can hear a difference. I have a $30 Behringer 202 which I thought sounded quite good.
Just make sure that if you do this there are no 'tells' between the 2 systems. e.g. if comparing pre amps and the volume changes at a different rate you can tell which is which, same for switch on thumps, relay clicks etc. You have to ensure that you really have no idea which is in use, it can be hard work to get there.
It's already been proved that long term is less reliable than fast switching, but more testing is always good so give it a go.
 

garbulky

Major Contributor
Joined
Feb 14, 2018
Messages
1,510
Likes
827
I have one main problem. My audio rig is used in the family area. And we are constnatly switching inputs and sources. The only time the same input would be used would be when I'm using only one specific source. So it would be hard for me not to know.
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
You don't have to level match. Or follow any protocol other than doing the testing blind. Have someone switch out one product for the other a dozen times once a day and you both keep score of which unit is what. Then at the 12th day compare notes and see how it matches your sighted evaluation.

I'm not following this at all.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
I'm not following this at all.

I think Amir is saying you could do everything as you normally do. No level matching. Listen as you want, do it over weeks. Develop an impression of the sound of a couple devices. Keeps notes. Except other than always knowing which you are listening to, you don't know. See if there is any consistency in your conclusion when the device is unknown.

I wouldn't consider it worth doing, but for those who insist there is some long term magic of impressionistic uncontrolled listening this would be a fair check on those impressions.

I long ago figured out (proved to myself), that if you don't match levels you are wasting your time. Secondly if you wait hours or days you are wasting your time. If nothing else background noises differ. Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
I think Amir is saying you could do everything as you normally do. No level matching. Listen as you want, do it over weeks. Develop an impression of the sound of a couple devices. Keeps notes. Except other than always knowing which you are listening to, you don't know. See if there is any consistency in your conclusion when the device is unknown.

I wouldn't consider it worth doing, but for those who insist there is some long term magic of impressionistic uncontrolled listening this would be a fair check on those impressions.

I long ago figured out (proved to myself), that if you don't match levels you are wasting your time. Secondly if you wait hours or days you are wasting your time. If nothing else background noises differ. Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.

No, I understand that. But, like you, I have doubts about what that would accomplish.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
No, I understand that. But, like you, I have doubts about what that would accomplish.
Well so many say the quick switch ABX method is too different from the normal way of listening that such blind testing has no relevance. Doing it the way Amir described would be acceptable to see if sound impressions were real or not without those complaints. The only thing left to say if such a test didn't match with sighted long term listening is "sight" is the primary difference. If there is a match, then it might vindicate those who say sighted long term listening has significant and real results that matter.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,598
Location
Central Fl
Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.
Absolutely. Living in the city I've always found that to be true. Late night listening always seemed to reveal more inner detail of the music and at a lower volume. Besides the background noise being there, or maybe because of that lack, there's an inner mental peace that just lets more of the music in. But then I've always been a night owl. :cool:
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
Well so many say the quick switch ABX method is too different from the normal way of listening that such blind testing has no relevance. Doing it the way Amir described would be acceptable to see if sound impressions were real or not without those complaints. The only thing left to say if such a test didn't match with sighted long term listening is "sight" is the primary difference. If there is a match, then it might vindicate those who say sighted long term listening has significant and real results that matter.

There's a reason they call it controlled testing.

Suppose, with one DUT, you never match the level of the other one over the course of time. (And with this method, you can almost guarantee you won't.) You're never comparing apples to apples. You can't conclude anything. And blind means blind. Not a person doing the switching who knows you, knows what DUT is playing when, etc. Hell, that's why the ABX comparator was built the way it was.

The results will be meaningless.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
There's a reason they call it controlled testing.

Suppose, with one DUT, you never match the level of the other one over the course of time. (And with this method, you can almost guarantee you won't.) You're never comparing apples to apples. You can't conclude anything. And blind means blind. Not a person doing the switching who knows you, knows what DUT is playing when, etc. Hell, that's why the ABX comparator was built the way it was.

The results will be meaningless.
I agree and I'm not an advocate for this method. I was explaining the thinking of Amir or so I believe.

My opinion is if you don't match levels you're wasting your time. Sighted or otherwise.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
No, I understand that. But, like you, I have doubts about what that would accomplish.
It will show that the sighted evaluations don't match the blind ones. It is the first step towards objectivity in audio. It only changes one variable: knowledge of equipment. If that invalides sighted conclusions, and the person internalizes that, it is a huge step forward for them.

It has the benefit of not deviating from what the listener normally does with their system.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
My opinion is if you don't match levels you're wasting your time. Sighted or otherwise.
If there is no immediate AB test, then there is no concept of level matching either. In normal listening we listen at all different volumes with varying content and levels. The person is not going to build up preference based on level in such a situation.

The overwhelming factor for for preference among subjectivist audio is the knowledge of what is being tested and method of testing. They bought a new gear, listen more intently and then hear more detail, air, etc. Take away that knowledge and this important factor is now eliminated. All else is left the same including long term listening, etc.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
I have one main problem. My audio rig is used in the family area. And we are constnatly switching inputs and sources. The only time the same input would be used would be when I'm using only one specific source. So it would be hard for me not to know.
Well then take our word for it that if you did the test blind, you would not remotely arrive at the same conclusions as you do sighted. :) I have done this test so many times.
 

Soniclife

Major Contributor
Forum Donor
Joined
Apr 13, 2017
Messages
4,499
Likes
5,417
Location
UK
It has the benefit of not deviating from what the listener normally does with their system.
There are a few things I'd like to be able to test like this, to really get to the bottom of if they are different, or its in my head, and to see how this methodology feels when you put yourself through it. Sadly the things I'm interested in are things like the old and new versions of my Devialets where I only own one version.

Blind testing codecs in foobar was a real wake up when I did it, highly recommended to anyone who never done something like it.
 

garbulky

Major Contributor
Joined
Feb 14, 2018
Messages
1,510
Likes
827
Well then take our word for it that if you did the test blind, you would not remotely arrive at the same conclusions as you do sighted. :) I have done this test so many times.
I remain skeptical, but I trust that you and others have conducted many blind tests whose results are meaninigful to you. Meanwhile I'm giving some new headphones a go. The HD 700's.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
I remain skeptical, but I trust that you and others have conducted many blind tests whose results are meaninigful to you.
The virtual reality of audio subjectivism is hard to let go. :)
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
It will show that the sighted evaluations don't match the blind ones. It is the first step towards objectivity in audio. It only changes one variable: knowledge of equipment. If that invalides sighted conclusions, and the person internalizes that, it is a huge step forward for them.

It has the benefit of not deviating from what the listener normally does with their system.

It might, it might not. But you are still comparing apples to zebras. And it doesn't change just "change one variable". Without level-matching, you are not comparing components tested under identical conditions. You are not validly comparing anything. You're really just diddling around.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
I've probably said this before, but I'll throw it out again. Let's start with the question being asked, "Can I arrive at meaningful results by doing long term listening rather than short comparisons?"

Start with some EQ. Put in a rise somewhere, maybe the top octave. Wherever. Do a rapid switching comparison while reducing the amount of EQ to determine the level at which you can't distinguish. If you have exceptionally sharp ears, it might be 0.1-0.2 dB, depending on where the EQ is. More normal ears can pick differences with very careful listening at 0.4-0.5 dB; I note that at these very subtle differences, the noted change might not sound like EQ, but a difference of some sort will be heard.

So let's say that for the chosen EQ, you find that you're good but not a prodigy and 0.4 dB is the smallest level you can distinguish. Let's call flat condition A, and EQed condition B. Get as familiar as you like with the two sounds until you're ready to start the test. Change volume as you normally would.

Have someone not in your presence make up a key sheet by doing 12 successive coin flips to generate a random table of A and B. Random.org is a good way to do this. I just ran twelve and got ABAAAAABBAAB. The person not in your presence will keep the key sheet until the end of the experiment.

When you're ready to begin the experiment, have a person not in your presence set the system condition. In this case, the first trial is A (make sure they go and do this even if there's no change and it's VITAL you not be present when that happens. Listen as long as you like, minutes, hours, days, whatever, changing volume at your desire, until you're ready to make a choice. Make your choice, record it, and have the system returned to A (or left in A, but the ritual still taking place so you don't know). When you're ready for the next guess, repeat this.

If you're persistent, when you're done, hand your score sheet to the keeper of the key sheet who will then calculate your score. At the end of your exhaustive months of long term listening and testing, you'll know if your acuity is better with rapid switching or long term listening. That will answer your question.

I would bet that if you make sure to keep things double blind, you'll not make it all the way through and figure out pretty quickly that the rapid method is far more sensitive.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
I've probably said this before, but I'll throw it out again. Let's start with the question being asked, "Can I arrive at meaningful results by doing long term listening rather than short comparisons?"

Start with some EQ. Put in a rise somewhere, maybe the top octave. Wherever. Do a rapid switching comparison while reducing the amount of EQ to determine the level at which you can't distinguish. If you have exceptionally sharp ears, it might be 0.1-0.2 dB, depending on where the EQ is. More normal ears can pick differences with very careful listening at 0.4-0.5 dB; I note that at these very subtle differences, the noted change might not sound like EQ, but a difference of some sort will be heard.

So let's say that for the chosen EQ, you find that you're good but not a prodigy and 0.4 dB is the smallest level you can distinguish. Let's call flat condition A, and EQed condition B. Get as familiar as you like with the two sounds until you're ready to start the test. Change volume as you normally would.

Have someone not in your presence make up a key sheet by doing 12 successive coin flips to generate a random table of A and B. Random.org is a good way to do this. I just ran twelve and got ABAAAAABBAAB. The person not in your presence will keep the key sheet until the end of the experiment.

When you're ready to begin the experiment, have a person not in your presence set the system condition. In this case, the first trial is A (make sure they go and do this even if there's no change and it's VITAL you not be present when that happens. Listen as long as you like, minutes, hours, days, whatever, changing volume at your desire, until you're ready to make a choice. Make your choice, record it, and have the system returned to A (or left in A, but the ritual still taking place so you don't know). When you're ready for the next guess, repeat this.

If you're persistent, when you're done, hand your score sheet to the keeper of the key sheet who will then calculate your score. At the end of your exhaustive months of long term listening and testing, you'll know if your acuity is better with rapid switching or long term listening. That will answer your question.

I would bet that if you make sure to keep things double blind, you'll not make it all the way through and figure out pretty quickly that the rapid method is far more sensitive.

No! Say it's not so.

My gut feelings, my wife in the kitchen, my friends, everything makes me feel so much more confident with a long term developed opinion of a bit of gear.

The pin prick to deflate that opinion however doing your experiment is so.........mean! I'd feel so much more emotionally attached just not doing this. There is more to life than simple truth. Factuality is so over-rated.
 
Top Bottom