Blind test design

[Dimitrov]

Hi guys,

Great place, glad to be here. I need some advice please.

I'm trying to set up an amp comparison event with some of my audiophile friends and the listening test is going to be blinded.

However I need some test design advice so that it can be a good test. I also need advice on the switch box. I've never undertaken anything like this, so I'm definitely out of my depth but I want to try.

So first things first, for switching the amplifiers, what would you recommend I use? I have a multimeter and as I understand it I just have to ensure the voltages are the same between both amplifiers using a test tone source at least 2-3 frequencies. Correct me if I'm wrong.

How should the test be set up? Should the participants first listen sighted, sans level matching and then blinded, level matched? Obviously there will be quite a time difference as I'll have to ensure the levels are matched. This takes time.

Do I let them choose the music? How many trials? I'm clueless when it comes to listening test design, so I'm all ears for advice on this. I'm not expecting lab quality results, just something that is far more reliable than the everyday casual listening evaluation.

If any of you have gone through this or better yet, have administered the test and have intricate details regarding how it should be done please post here. Thanks!

 

[Thomas savage]

Hi guys,

Great place, glad to be here. I need some advice please.

I'm trying to set up an amp comparison event with some of my audiophile friends and the listening test is going to be blinded.

However I need some test design advice so that it can be a good test. I also need advice on the switch box. I've never undertaken anything like this, so I'm definitely out of my depth but I want to try.

So first things first, for switching the amplifiers, what would you recommend I use? I have a multimeter and as I understand it I just have to ensure the voltages are the same between both amplifiers using a test tone source at least 2-3 frequencies. Correct me if I'm wrong.

How should the test be set up? Should the participants first listen sighted, sans level matching and then blinded, level matched? Obviously there will be quite a time difference as I'll have to ensure the levels are matched. This takes time.

Do I let them choose the music? How many trials? I'm clueless when it comes to listening test design, so I'm all ears for advice on this. I'm not expecting lab quality results, just something that is far more reliable than the everyday casual listening evaluation.

If any of you have gone through this or better yet, have administered the test and have intricate details regarding how it should be done please post here. Thanks!

A warm Welcome to you and thanks for taking the time to contribute

 

[Purité Audio]

I would definitely level match using a test tone say 400Hz and mini volt meter, are you going to be comparing power amps, that is going to be trickier because of the delay caused by plugging /unplugging.
You also need to make sure that both amplifiers are capable of properly driving the loudspeakers, if one is going to clip then obviously they will sound different.
Easier to compare DACs through the same preamp, again level matched.
https://www.puriteaudio.co.uk/single-post/2017/02/08/Level-matching-for-fun

Let us know how you get on.
Keith

 

[Dimitrov]

Thanks Keith. Wow, that was quick!

I wasn't expecting to compare power amps, just integrated amps and AV receivers as the latter seems to get a lot of hate and derision from audiophiles. However now that you bring it up, I've always wanted to compare an AV receiver vs one with a power amp.

I'm not convinced that simply raising the total potentially available power has any audible effect on the sound which is what audiophiles often like to claim. If I compare both at reasonable volume levels I expect them to sound very similar at least. However I would love to test this all out. Hate being on the sidelines and reading about it, would like to experience it for myself.

My friends are providing the speakers. Audiophile-grade brand, I think a B&W model.

 

[Dimitrov]

I read your article. Very informative!

 

[Dimitrov]

A warm Welcome to you and thanks for taking the time to contribute

Thanks Thomas for the warm welcome!

 

[Purité Audio]

My pleasure I am not an expert by any means, what is going to be difficult is the time taken to disconnect the amps and connect the next, ideallytwo sets of identical speakers!
Also have a look at the impedance plot of the speakers, often there is very low impedence within the speakers Fr which amps find more difficult to drive.
As you said I believe that is why some listeners report differences,simply because the amps can't drive the loudspeakers properly.
Keith

 

[Dimitrov]

My pleasure I am not an expert by any means, what is going to be difficult is the time taken to disconnect the amps and connect the next, ideallytwo sets of identical speakers!
Also have a look at the impedance plot of the speakers, often there is very low impedence within the speakers Fr which amps find more difficult to drive.
As you said I believe that is why some listeners report differences,simply because the amps can't drive the loudspeakers properly.
Keith

Well this is why I was hoping there might be a amp switch box that could switch between two amps. Speakers tested will be the same throughout the test. Have no idea what type of switch box would work for this kind of test.

My friends swear that an AV receiver can't drive their 683 floor standers, and I see nothing about their impedance plot that would indicate such a thing. Certainly not the easiest speaker to drive, but I think there is a stigma/paranoia attached to B&W speakers as being "very difficult" to drive.

I have no doubt that a bigger amp can drive them to higher levels without audible distress but the key is whether driving them at more reasonable levels would result in any "very noticeable" or "night and day" differences. This is just one of the claims I want to test out.

 

[amirm]

So first things first, for switching the amplifiers, what would you recommend I use? I have a multimeter and as I understand it I just have to ensure the voltages are the same between both amplifiers using a test tone source at least 2-3 frequencies. Correct me if I'm wrong.

First welcome to the forum. And importantly wanting to take an initiative to do such tests. Highly commended!

This was answered before. Simply play a tone and measure the output voltage on the speaker terminals. Please be careful to not short them out of course as that is easy to do with voltmeter leads.

How should the test be set up? Should the participants first listen sighted, sans level matching and then blinded, level matched? Obviously there will be quite a time difference as I'll have to ensure the levels are matched. This takes time.

Depends on whether you want real data or just prove the other side wrong. . Assuming it is the former:

1. You should let listeners play with the setup as much as they want. This can be sighted or blind.
2. Put in a control. For example, you could set the levels audibly higher on one unit then run a blind test to see if people can tell the difference. That demonstrates that the entire system is running correctly. If for example there is a mistake in switching or scoring, this will catch those.
3. For the real test, use your AVR gain settings and change those up and down to match the integrated. Otherwise you have to get a multi-channel pre-amp and set the gains for each amp. That brings up criticism that this added device hides the differences.

Do I let them choose the music?

Again, it depends on whether you want to prove them wrong or get good data. For the former, yes, for sure let them pick their music. For the latter, the main difference I find with amps is when they have been pushed to the limit with bass. So I pick drum heavy tracks and keep turning up the volume and listen for bass thinning out. If that happens at all.

How many trials?

Here is the simple math. If you run it once, then probability of guessing is 50%. The goal is to get this down to 5%. If you do it a second time, the probability is 25% and so on. At about 8 times or so, you get to 95% confidence assuming the person gets all the answers right. If not, you have to keep continuing. See this article I wrote: http://audiosciencereview.com/forum/index.php?threads/statistics-of-abx-testing.170/

My guess is that your listeners will fail at a high rate which means that you can stop after five or six switches when it becomes clear they are never going to reach 95% confidence.

I'm clueless when it comes to listening test design, so I'm all ears for advice on this. I'm not expecting lab quality results, just something that is far more reliable than the everyday casual listening evaluation.

Doing such tests is difficult and time consuming. Kudos for trying though. Here is the only published blind test of this type: http://audiosciencereview.com/forum...ts-did-show-amplifiers-to-sound-different.23/

Note that one of the authors told me after the fact that they could hear the one amp breaking up or something before deciding to do a blind test. If so, it is a corrupt experiment but still, might serve as a good guide for you on protocol.

Again, welcome to the forum.

 

[Dimitrov]

First welcome to the forum. And importantly wanting to take an initiative to do such tests. Highly commended!

This was answered before. Simply play a tone and measure the output voltage on the speaker terminals. Please be careful to not short them out of course as that is easy to do with voltmeter leads.


Depends on whether you want real data or just prove the other side wrong. . Assuming it is the former:

1. You should let listeners play with the setup as much as they want. This can be sighted or blind.
2. Put in a control. For example, you could set the levels audibly higher on one unit then run a blind test to see if people can tell the difference. That demonstrates that the entire system is running correctly. If for example there is a mistake in switching or scoring, this will catch those.
3. For the real test, use your AVR gain settings and change those up and down to match the integrated. Otherwise you have to get a multi-channel pre-amp and set the gains for each amp. That brings up criticism that this added device hides the differences.


Again, it depends on whether you want to prove them wrong or get good data. For the former, yes, for sure let them pick their music. For the latter, the main difference I find with amps is when they have been pushed to the limit with bass. So I pick drum heavy tracks and keep turning up the volume and listen for bass thinning out. If that happens at all.


Here is the simple math. If you run it once, then probability of guessing is 50%. The goal is to get this down to 5%. If you do it a second time, the probability is 25% and so on. At about 8 times or so, you get to 95% confidence assuming the person gets all the answers right. If not, you have to keep continuing. See this article I wrote: http://audiosciencereview.com/forum/index.php?threads/statistics-of-abx-testing.170/

My guess is that your listeners will fail at a high rate which means that you can stop after five or six switches when it becomes clear they are never going to reach 95% confidence.


Doing such tests is difficult and time consuming. Kudos for trying though. Here is the only published blind test of this type: http://audiosciencereview.com/forum...ts-did-show-amplifiers-to-sound-different.23/

Note that one of the authors told me after the fact that they could hear the one amp breaking up or something before deciding to do a blind test. If so, it is a corrupt experiment but still, might serve as a good guide for you on protocol.

Again, welcome to the forum.

Wow, thanks Amirm! Very informative post. I may decide to pick your brain on a few details.

Since you seem to know a whole lot about this stuff, any pointers regarding an amp switching box? I've heard some people use speaker selector boxes and modify them, but I'm not sure about this.

Disclaimer - I do want to prove my audiophile friends wrong, or at least humble them a little. Makes for interesting conversation too.

 

[Purité Audio]

I am not sure an amplifier switching box exists, you need to move the speaker cables from one amp to another, unless you can have two complete systems and switch from one to another , a DAC test'comparison would be easier as a first attempt.
Keith

 

[amirm]

My pleasure. On your question, do both amplifiers have protection circuits? I know the AVR does but check the other amp. The reason I ask is that if you suddenly disconnect a speaker from an amp, it may create a spike that damages the output stages. If they are robust in that manner, i.e. protect themselves, then a speaker selector box used in reverse would do that. You would then have a splitter that would feed both amps simultaneously.