• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Blind test - objectivists with tin hearing?

garbulky

Addicted to Fun and Learning
Joined
Feb 14, 2018
Messages
828
Likes
227
You don't have to level match. Or follow any protocol other than doing the testing blind. Have someone switch out one product for the other a dozen times once a day and you both keep score of which unit is what. Then at the 12th day compare notes and see how it matches your sighted evaluation.
You may have something there....
I may try such a thing. I could use my harmony universal remote so that I wouldn't know which preamp I was using. I'd just need another DAC, perhaps one I find not as pleasing/different like the Oppo 205.Well actually since my DC-1 is an analog preamp, I could just that as the input for a cheap dac and see if I can hear a difference. I have a $30 Behringer 202 which I thought sounded quite good.
 

Soniclife

Major Contributor
Joined
Apr 13, 2017
Messages
1,336
Likes
832
Location
UK
You may have something there....
I may try such a thing. I could use my harmony universal remote so that I wouldn't know which preamp I was using. I'd just need another DAC, perhaps one I find not as pleasing/different like the Oppo 205.Well actually since my DC-1 is an analog preamp, I could just that as the input for a cheap dac and see if I can hear a difference. I have a $30 Behringer 202 which I thought sounded quite good.
Just make sure that if you do this there are no 'tells' between the 2 systems. e.g. if comparing pre amps and the volume changes at a different rate you can tell which is which, same for switch on thumps, relay clicks etc. You have to ensure that you really have no idea which is in use, it can be hard work to get there.
It's already been proved that long term is less reliable than fast switching, but more testing is always good so give it a go.
 

garbulky

Addicted to Fun and Learning
Joined
Feb 14, 2018
Messages
828
Likes
227
I have one main problem. My audio rig is used in the family area. And we are constnatly switching inputs and sources. The only time the same input would be used would be when I'm using only one specific source. So it would be hard for me not to know.
 

jsrtheta

Active Member
Joined
May 20, 2018
Messages
205
Likes
144
You don't have to level match. Or follow any protocol other than doing the testing blind. Have someone switch out one product for the other a dozen times once a day and you both keep score of which unit is what. Then at the 12th day compare notes and see how it matches your sighted evaluation.
I'm not following this at all.
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
5,983
Likes
5,218
I'm not following this at all.
I think Amir is saying you could do everything as you normally do. No level matching. Listen as you want, do it over weeks. Develop an impression of the sound of a couple devices. Keeps notes. Except other than always knowing which you are listening to, you don't know. See if there is any consistency in your conclusion when the device is unknown.

I wouldn't consider it worth doing, but for those who insist there is some long term magic of impressionistic uncontrolled listening this would be a fair check on those impressions.

I long ago figured out (proved to myself), that if you don't match levels you are wasting your time. Secondly if you wait hours or days you are wasting your time. If nothing else background noises differ. Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.
 

jsrtheta

Active Member
Joined
May 20, 2018
Messages
205
Likes
144
I think Amir is saying you could do everything as you normally do. No level matching. Listen as you want, do it over weeks. Develop an impression of the sound of a couple devices. Keeps notes. Except other than always knowing which you are listening to, you don't know. See if there is any consistency in your conclusion when the device is unknown.

I wouldn't consider it worth doing, but for those who insist there is some long term magic of impressionistic uncontrolled listening this would be a fair check on those impressions.

I long ago figured out (proved to myself), that if you don't match levels you are wasting your time. Secondly if you wait hours or days you are wasting your time. If nothing else background noises differ. Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.
No, I understand that. But, like you, I have doubts about what that would accomplish.
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
5,983
Likes
5,218
No, I understand that. But, like you, I have doubts about what that would accomplish.
Well so many say the quick switch ABX method is too different from the normal way of listening that such blind testing has no relevance. Doing it the way Amir described would be acceptable to see if sound impressions were real or not without those complaints. The only thing left to say if such a test didn't match with sighted long term listening is "sight" is the primary difference. If there is a match, then it might vindicate those who say sighted long term listening has significant and real results that matter.
 

Sal1950

Major Contributor
The Chicago Crusher
Joined
Mar 1, 2016
Messages
5,378
Likes
2,705
Location
Central Fl
Having lived near a busy highway once I would experience the following. Late at night with no one being bothered I'd listen at a level I was very happy with. I could hear the music very well. Having changed absolutely nothing including volume, the next afternoon the music was too quiet to enjoy and you couldn't hear details you simply couldn't enjoy the music. The difference? Traffic had raised the noise floor by about 20 db vs night time no traffic listening.
Absolutely. Living in the city I've always found that to be true. Late night listening always seemed to reveal more inner detail of the music and at a lower volume. Besides the background noise being there, or maybe because of that lack, there's an inner mental peace that just lets more of the music in. But then I've always been a night owl. :cool:
 

jsrtheta

Active Member
Joined
May 20, 2018
Messages
205
Likes
144
Well so many say the quick switch ABX method is too different from the normal way of listening that such blind testing has no relevance. Doing it the way Amir described would be acceptable to see if sound impressions were real or not without those complaints. The only thing left to say if such a test didn't match with sighted long term listening is "sight" is the primary difference. If there is a match, then it might vindicate those who say sighted long term listening has significant and real results that matter.
There's a reason they call it controlled testing.

Suppose, with one DUT, you never match the level of the other one over the course of time. (And with this method, you can almost guarantee you won't.) You're never comparing apples to apples. You can't conclude anything. And blind means blind. Not a person doing the switching who knows you, knows what DUT is playing when, etc. Hell, that's why the ABX comparator was built the way it was.

The results will be meaningless.
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
5,983
Likes
5,218
There's a reason they call it controlled testing.

Suppose, with one DUT, you never match the level of the other one over the course of time. (And with this method, you can almost guarantee you won't.) You're never comparing apples to apples. You can't conclude anything. And blind means blind. Not a person doing the switching who knows you, knows what DUT is playing when, etc. Hell, that's why the ABX comparator was built the way it was.

The results will be meaningless.
I agree and I'm not an advocate for this method. I was explaining the thinking of Amir or so I believe.

My opinion is if you don't match levels you're wasting your time. Sighted or otherwise.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
19,507
Likes
20,620
Location
Seattle Area
No, I understand that. But, like you, I have doubts about what that would accomplish.
It will show that the sighted evaluations don't match the blind ones. It is the first step towards objectivity in audio. It only changes one variable: knowledge of equipment. If that invalides sighted conclusions, and the person internalizes that, it is a huge step forward for them.

It has the benefit of not deviating from what the listener normally does with their system.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
19,507
Likes
20,620
Location
Seattle Area
My opinion is if you don't match levels you're wasting your time. Sighted or otherwise.
If there is no immediate AB test, then there is no concept of level matching either. In normal listening we listen at all different volumes with varying content and levels. The person is not going to build up preference based on level in such a situation.

The overwhelming factor for for preference among subjectivist audio is the knowledge of what is being tested and method of testing. They bought a new gear, listen more intently and then hear more detail, air, etc. Take away that knowledge and this important factor is now eliminated. All else is left the same including long term listening, etc.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
19,507
Likes
20,620
Location
Seattle Area
I have one main problem. My audio rig is used in the family area. And we are constnatly switching inputs and sources. The only time the same input would be used would be when I'm using only one specific source. So it would be hard for me not to know.
Well then take our word for it that if you did the test blind, you would not remotely arrive at the same conclusions as you do sighted. :) I have done this test so many times.
 

Soniclife

Major Contributor
Joined
Apr 13, 2017
Messages
1,336
Likes
832
Location
UK
It has the benefit of not deviating from what the listener normally does with their system.
There are a few things I'd like to be able to test like this, to really get to the bottom of if they are different, or its in my head, and to see how this methodology feels when you put yourself through it. Sadly the things I'm interested in are things like the old and new versions of my Devialets where I only own one version.

Blind testing codecs in foobar was a real wake up when I did it, highly recommended to anyone who never done something like it.
 

garbulky

Addicted to Fun and Learning
Joined
Feb 14, 2018
Messages
828
Likes
227
Well then take our word for it that if you did the test blind, you would not remotely arrive at the same conclusions as you do sighted. :) I have done this test so many times.
I remain skeptical, but I trust that you and others have conducted many blind tests whose results are meaninigful to you. Meanwhile I'm giving some new headphones a go. The HD 700's.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
19,507
Likes
20,620
Location
Seattle Area
I remain skeptical, but I trust that you and others have conducted many blind tests whose results are meaninigful to you.
The virtual reality of audio subjectivism is hard to let go. :)
 

jsrtheta

Active Member
Joined
May 20, 2018
Messages
205
Likes
144
It will show that the sighted evaluations don't match the blind ones. It is the first step towards objectivity in audio. It only changes one variable: knowledge of equipment. If that invalides sighted conclusions, and the person internalizes that, it is a huge step forward for them.

It has the benefit of not deviating from what the listener normally does with their system.
It might, it might not. But you are still comparing apples to zebras. And it doesn't change just "change one variable". Without level-matching, you are not comparing components tested under identical conditions. You are not validly comparing anything. You're really just diddling around.
 

SIY

Major Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
2,459
Likes
4,041
Location
Phoenix, AZ
I've probably said this before, but I'll throw it out again. Let's start with the question being asked, "Can I arrive at meaningful results by doing long term listening rather than short comparisons?"

Start with some EQ. Put in a rise somewhere, maybe the top octave. Wherever. Do a rapid switching comparison while reducing the amount of EQ to determine the level at which you can't distinguish. If you have exceptionally sharp ears, it might be 0.1-0.2 dB, depending on where the EQ is. More normal ears can pick differences with very careful listening at 0.4-0.5 dB; I note that at these very subtle differences, the noted change might not sound like EQ, but a difference of some sort will be heard.

So let's say that for the chosen EQ, you find that you're good but not a prodigy and 0.4 dB is the smallest level you can distinguish. Let's call flat condition A, and EQed condition B. Get as familiar as you like with the two sounds until you're ready to start the test. Change volume as you normally would.

Have someone not in your presence make up a key sheet by doing 12 successive coin flips to generate a random table of A and B. Random.org is a good way to do this. I just ran twelve and got ABAAAAABBAAB. The person not in your presence will keep the key sheet until the end of the experiment.

When you're ready to begin the experiment, have a person not in your presence set the system condition. In this case, the first trial is A (make sure they go and do this even if there's no change and it's VITAL you not be present when that happens. Listen as long as you like, minutes, hours, days, whatever, changing volume at your desire, until you're ready to make a choice. Make your choice, record it, and have the system returned to A (or left in A, but the ritual still taking place so you don't know). When you're ready for the next guess, repeat this.

If you're persistent, when you're done, hand your score sheet to the keeper of the key sheet who will then calculate your score. At the end of your exhaustive months of long term listening and testing, you'll know if your acuity is better with rapid switching or long term listening. That will answer your question.

I would bet that if you make sure to keep things double blind, you'll not make it all the way through and figure out pretty quickly that the rapid method is far more sensitive.
 

Blumlein 88

Major Contributor
Joined
Feb 23, 2016
Messages
5,983
Likes
5,218
I've probably said this before, but I'll throw it out again. Let's start with the question being asked, "Can I arrive at meaningful results by doing long term listening rather than short comparisons?"

Start with some EQ. Put in a rise somewhere, maybe the top octave. Wherever. Do a rapid switching comparison while reducing the amount of EQ to determine the level at which you can't distinguish. If you have exceptionally sharp ears, it might be 0.1-0.2 dB, depending on where the EQ is. More normal ears can pick differences with very careful listening at 0.4-0.5 dB; I note that at these very subtle differences, the noted change might not sound like EQ, but a difference of some sort will be heard.

So let's say that for the chosen EQ, you find that you're good but not a prodigy and 0.4 dB is the smallest level you can distinguish. Let's call flat condition A, and EQed condition B. Get as familiar as you like with the two sounds until you're ready to start the test. Change volume as you normally would.

Have someone not in your presence make up a key sheet by doing 12 successive coin flips to generate a random table of A and B. Random.org is a good way to do this. I just ran twelve and got ABAAAAABBAAB. The person not in your presence will keep the key sheet until the end of the experiment.

When you're ready to begin the experiment, have a person not in your presence set the system condition. In this case, the first trial is A (make sure they go and do this even if there's no change and it's VITAL you not be present when that happens. Listen as long as you like, minutes, hours, days, whatever, changing volume at your desire, until you're ready to make a choice. Make your choice, record it, and have the system returned to A (or left in A, but the ritual still taking place so you don't know). When you're ready for the next guess, repeat this.

If you're persistent, when you're done, hand your score sheet to the keeper of the key sheet who will then calculate your score. At the end of your exhaustive months of long term listening and testing, you'll know if your acuity is better with rapid switching or long term listening. That will answer your question.

I would bet that if you make sure to keep things double blind, you'll not make it all the way through and figure out pretty quickly that the rapid method is far more sensitive.
No! Say it's not so.

My gut feelings, my wife in the kitchen, my friends, everything makes me feel so much more confident with a long term developed opinion of a bit of gear.

The pin prick to deflate that opinion however doing your experiment is so.........mean! I'd feel so much more emotionally attached just not doing this. There is more to life than simple truth. Factuality is so over-rated.
 
Top Bottom