Double Blind tests *did* show amplifiers to sound different

[SIY]

I conducted listening tests where I didn't tell participants what was being changed or altered, but just asked them their impressions. Although I knew what was going on, realistically the way it was set up, there was no way for participants to know what, if anything (and in many instances I did nothing) was changing or what they were listening for. In all instances, I was far away from the amplifier and out of eyesight. The participants had no in depth knowledge of the R&D I was doing, what my specific goals were, and how I was doing it. The bias of the amplifier in question was varied to several levels, and the result was in 100% uniformity for the bias being at one particular setting. I had done no AP measurements beforehand; I was just pulling bias levels essentially out of the air. After these listening sessions, I had AP measurements taken of distortion at at various power levels from 20kHz down to 20Hz. The results were unambiguous; the preferred bias level was exactly in the "sweet" zone of minimum distortion across the audio band where distortion was minimized.

Have you considered doing a listening test with actual double blind controls to validate (or refute) your hypothesis?

 

[MakeMineVinyl]

Have you considered doing a listening test with actual double blind controls to validate (or refute) your hypothesis?

We don't have the switching gear to do that, and we only had one prototype amp anyway. The test at any rate told me what I needed to know.

 

[SIY]

We don't have the switching gear to do that, and we only had one prototype amp anyway. The test at any rate told me what I needed to know.

It's not too hard to do without switching gear. If you have a sincere interest in getting valid listening test results, I'd be more than happy to help you set up proper unbiased (pun unintentional) testing.

 

[MakeMineVinyl]

It's not too hard to do without switching gear. If you have a sincere interest in getting valid listening test results, I'd be more than happy to help you set up proper unbiased (pun unintentional) testing.

Thanks for the offer. If I do listing test like this in the future I will certainly let you know. The other engineers which do listening tests most always listen to just our product by itself and only one product at a time. Changing their methods might be like herding cats.

Could anything more have been done with the bias testing scenario I explained given that I only had the one prototype? If there's a better mythology I could have used within those restrictions I'd be interested in those.

 

[solderdude]

The other engineers which do listening tests most always listen to just our product by itself and only one product at a time. Changing their methods might be like herding cats.

That's what S. Olive also ran into. He made them aware they could improve on their testing.

 

[Feanor]

Its really not that difficult to at least;
1. Match volumes accurately
2. Not know what product you are listening to.

If people did that then they would be taken seriously with their input. Just go over to somewhere like Whats Best forum to see the amount of complete drivel some people talk about their uncontrolled subjective findings.

This is a science based forum so I have no compunction about telling people to "go away" as you put it if they dont understand or are unwilling to accept the basic science and issues related to cognitive bias.

... The hubris of know-it-all science -- actually not very scientific.

 

[solderdude]

We're a much smaller company than Harmon (was also an engineer at JBL), and I think our cats are pretty resistant to change.

Time to let a viscious dog loose in the house

 

[March Audio]

... The hubris of know-it-all science -- actually not very scientific.

That's possibly one of the most ridiculous statements I have seen recently on this forum.

It's clear you simply don't understand, or don't want to accept the proven science and impact of cognitive bias.

 

[March Audio]

I was initially going to ignore this, but realistically I believe I can give some insight to some practical results on a product I'm working on now. Specifically, I designed a system which regulates bias current in real time according to several parameters. Since bias is regulated, it can be held at a significantly higher level in an optimum zone for least crossover distortion, and by inference, listening quality.

I conducted listening tests where I didn't tell participants what was being changed or altered, but just asked them their impressions. Although I knew what was going on, realistically the way it was set up, there was no way for participants to know what, if anything (and in many instances I did nothing) was changing or what they were listening for. In all instances, I was far away from the amplifier and out of eyesight. The participants had no in depth knowledge of the R&D I was doing, what my specific goals were, and how I was doing it. The bias of the amplifier in question was varied to several levels, and the result was in 100% uniformity for the bias being at one particular setting. I had done no AP measurements beforehand; I was just pulling bias levels essentially out of the air. After these listening sessions, I had AP measurements taken of distortion at at various power levels from 20kHz down to 20Hz. The results were unambiguous; the preferred bias level was exactly in the "sweet" zone of minimum distortion across the audio band where distortion was minimized.

However this bias level was also far higher (like about 200% higher) than could practically be set in an 8 channel home theater amplifier using conventional methods of setting bias, i.e not strictly regulated under all operating conditions. This was overcome so optimum bias could be set, with no downsides.

Also, to your point, this was obvious in the subsequent AP measurements. I purposefully did the subjective tests first to eliminate as much as practical any confirmation bias I might have had other than what was my design goal for the system.

So yes, audible differences proved to be real, and yes they were measurable. However in this case I did things in reverse to both test my design and to eliminate as much as possible any validation bias that what I was doing was correct - or not.

To address your post more specifically, the answer is yes, I have heard differences in other amplifiers. It has been my general experience that although amplifiers might measure "the same" static distortion, spectral analysis of the "worse" sounding amplifiers reveals higher levels of upper harmonic distortion, where the "better" sounding ones have mostly lower order (and specifically even order) distortion products. So this seems to validate both measuring and listening, which is what I do.

What isn't revealed in any measurements are (as I have posted repeatedly before) non-fidelity issues like if the amplifier makes pops or other noises upon turn-on or turn off. Also in the system I'm working with now, since it is an innovative way of accomplishing bias control, does this result in any transient noises or other artifacts? Attempting to do this by sitting by a 'scope, hoping to "see" a transient event would be very inefficient, and would not subjectively tell me anything about what any such event actually sounded like; was it an oscillation, a pop, a hiss, a sliding tone or any other artifact? If any of these things were heard (they weren't), the nature of them would have given a clue of what to look for in subsequent troubleshooting.

Really, listening for abnormal behavior is a routine procedure in the development process even in more conventional amplifiers. This is not generally talked about because it isn't "sexy" or controversial, but is is a very large part of listening to new designs. Although I don't have any specific experience on this, I suspect that a lot of listening was carried out in the development of class "D" amplifier modules.

So best measurements correlated with best sound.

Of course measurements are relevant in all aspects of amp behaviour. They also allow you to see things you might not easily notice by ear.

No one was or is suggesting that measurements are the only diagnostic tool or that listening shouldn't be used, that's your misinterpretation.

 

[SIY]

Could anything more have been done with the bias testing scenario I explained given that I only had the one prototype? If there's a better mythology I could have used within those restrictions I'd be interested in those.

Yes. And my offer is a sincere one- you’re not that far away, and I can pay for the trip by writing an article about ATI and some of the things going on there, if you’re willing to give me a tour. And to sweeten the deal, if you want, I’ll give a presentation to the recalcitrant engineers to try to make things easier in that respect.

 

[oivavoi]

I was initially going to ignore this, but realistically I believe I can give some insight to some practical results on a product I'm working on now. Specifically, I designed a system which regulates bias current in real time according to several parameters. Since bias is regulated, it can be held at a significantly higher level in an optimum zone for least crossover distortion, and by inference, listening quality.

I conducted listening tests where I didn't tell participants what was being changed or altered, but just asked them their impressions. Although I knew what was going on, realistically the way it was set up, there was no way for participants to know what, if anything (and in many instances I did nothing) was changing or what they were listening for. In all instances, I was far away from the amplifier and out of eyesight. The participants had no in depth knowledge of the R&D I was doing, what my specific goals were, and how I was doing it. The bias of the amplifier in question was varied to several levels, and the result was in 100% uniformity for the bias being at one particular setting. I had done no AP measurements beforehand; I was just pulling bias levels essentially out of the air. After these listening sessions, I had AP measurements taken of distortion at at various power levels from 20kHz down to 20Hz. The results were unambiguous; the preferred bias level was exactly in the "sweet" zone of minimum distortion across the audio band where distortion was minimized.

However this bias level was also far higher (like about 200% higher) than could practically be set in an 8 channel home theater amplifier using conventional methods of setting bias, i.e not strictly regulated under all operating conditions. This was overcome so optimum bias could be set, with no downsides.

Also, to your point, this was obvious in the subsequent AP measurements. I purposefully did the subjective tests first to eliminate as much as practical any confirmation bias I might have had other than what was my design goal for the system.

So yes, audible differences proved to be real, and yes they were measurable. However in this case I did things in reverse to both test my design and to eliminate as much as possible any validation bias that what I was doing was correct - or not.

To address your post more specifically, the answer is yes, I have heard differences in other amplifiers. It has been my general experience that although amplifiers might measure "the same" static distortion, spectral analysis of the "worse" sounding amplifiers reveals higher levels of upper harmonic distortion, where the "better" sounding ones have mostly lower order (and specifically even order) distortion products. So this seems to validate both measuring and listening, which is what I do.

What isn't revealed in any measurements are (as I have posted repeatedly before) non-fidelity issues like if the amplifier makes pops or other noises upon turn-on or turn off. Also in the system I'm working with now, since it is an innovative way of accomplishing bias control, does this result in any transient noises or other artifacts? Attempting to do this by sitting by a 'scope, hoping to "see" a transient event would be very inefficient, and would not subjectively tell me anything about what any such event actually sounded like; was it an oscillation, a pop, a hiss, a sliding tone or any other artifact? If any of these things were heard (they weren't), the nature of them would have given a clue of what to look for in subsequent troubleshooting.

Really, listening for abnormal behavior is a routine procedure in the development process even in more conventional amplifiers. This is not generally talked about because it isn't "sexy" or controversial, but is is a very large part of listening to new designs. Although I don't have any specific experience on this, I suspect that a lot of listening was carried out in the development of class "D" amplifier modules.

Very interesting!

I'll just echo the sentiment that I hope you don't quit the forum. It's great to get input into how development etc happens at a reputable company like ATI.

 

[Feanor]

That's possibly one of the most ridiculous statements I have seen recently on this forum.

It's clear you simply don't understand, or don't want to accept the proven science and impact of cognitive bias.

... QED

 

[March Audio]

... QED

Then I will leave you to your faith and dogma. Personally I will go with the proven science

The anti science atitude apparant in many areas is a very interesting and curious cultural phenomenon.

 

[MakeMineVinyl]

So best measurements correlated with best sound.

Of course measurements are relevant in all aspects of amp behaviour. They also allow you to see things you might not easily notice by ear.

I don't disagree with any of that at all. My only point was that when trying to find some non-typical behavior, its sometimes simpler and more direct to start by just listening to it. There's also some sounds to listen for which are purely mechanical like buzzes, hums and fan noise where applicable.

 

[Feanor]

Then I will leave you to your faith and dogma. Personally I will go with the proven science

The anti science atitude apparant in many areas is a very interesting and curious cultural phenomenon.

Hearing differences between amplifiers isn't like alien abduction. The latter is attested to be a hand full of cranks; the former by hundreds of thousands of audiophiles worldwide.

I'm not anti-science. Good scientists know that all science is subject to revision. It's not us audiophiles who are the dogmatists, it's the-all-amps-sound-alike who are the dogmatists. (It seems to me it's the engineers who believe that science, their branch at least, is cast in stone).

I come to ASR because I believe that the differing measurements between, say, a Benchmark and a typical AVR are important to the sound. It's hard to understand why they would matter to an all amps sound the same dogmatist.

 

[March Audio]

Hearing differences between amplifiers isn't like alien abduction. The latter is attested to be a hand full of cranks; the former by hundreds of thousands of audiophiles worldwide.

I'm not anti-science. Good scientists know that all science is subject to revision. It's not us audiophiles who are the dogmatists, it's the-all-amps-sound-alike dogmatists. (It seems to me it's the engineers who believe that science, their branch at least, is cast in stone).

Don't shift the goalposts. That's not what we were talking about. The question was the validity of uncontrolled sighted comparisons not if it is possible for there to be differences in the sound of amplifiers, which has already been agreed with. No one has said all amps sound alike.

Go back and read what I have said, we have been over all of this and I am not going to waste my time going over it again.

However thousands of audiophiles worldwide do precisely the wrong things and end up with faulty conclusions. As I previously said, take a trip over to What's Best forum if you want to see some of the subjective drivel discussed by some audiophiles.

You also seem to miss the point somewhat. People who offer up opinions on sound gained through uncontrolled means could actually be correct in their comments, they could well be hearing real differences and accurately describing them. The problem is that we can't trust those views because the risk of cognitive bias is so high. So the data and comments becomes worthless because we have no confidence in it being correct.

All this is proven science, but as previously mentioned you can believe whatever you wish.

 

[NTK]

FWIW, below is from a post by Dr Toole at AVSForum:

tnargs said "Audiophiles tend to struggle to accept how little credence dbt experts give to 'amp sound' that is so important to sighted listening experts."

Truth is that many years ago I thought that amps had distinctive sounds - until I did a proper level matched blind test. I had just purchased what was then called a "super amp" - something with more power output than was practical with tubes and first gen solid state stuff. I spent hours "hearing" things I had never heard before, feeling pride of ownership and of technical advances.

However because of who I am I had to test it properly. In precisely level matched instantaneously switched A/B/C/D blind tests, including my old amp and several new ones, nobody could hear any large differences - including me - and I let the test run for many days, inviting local audiophiles and audio salesmen in, returning to it, listening for long periods. I was really surprised, as were the retailers who routinely demonstrated "differences" in their stores! A couple of them had been participating in loudspeaker evaluations as well, and they stopped coming to my evaluations because - and they were honest in this - it got in the way of selling products. I absolutely understood.

Small differences that were heard were traceable to small differences in frequency response and, in one case, a faulty overload protection circuit (the manufacturer did a design revision when I told them). One of the amps was a pure class B, with crossover distortion that could easily be heard with pure tones but which was not noticed with most music - it was with some, though. It had been done deliberately for all the wrong reasons. Back then some early generation solid state amps did have some problems within the audio bandwidth. Tube amps were easy to spot because they changed the frequency response of the loudspeakers (high output impedance). So, count me among those that have learned to trust: (a) blind tests and (b) the right measurements.

A subsequent test done for a Canadian audio magazine was a comparison of small to large receivers. The differences were hard to hear, up to clipping or overload protection. Current limitations when driving low impedances was a problem. However, one manufacturer's product line showed a clear progression of rising sound quality scores with price. The reason was easy to find: they simply readjusted the "zero" settings on the tone controls so that lower priced units exhibited progressively more rolled off bass and treble. I would like to think that those days are behind us.

But, I fully understand that in the world of audio reviewing, marketing, etc. all of this "science" does get in the way of a good story ​

​

​

​

 

[Sal1950]

It's not us audiophiles who are the dogmatists, it's the-all-amps-sound-alike who are the dogmatists. (It seems to me it's the engineers who believe that science, their branch at least, is cast in stone).

 

[MakeMineVinyl]

Auto-biasing has always interested me. I tried it once or twice way back when I was piddling with audio designs but too often it added some sort of noise or distortion that was audible. Measuring it was harder since it was dynamic and we didn't have fancy digital analyzers and DSOs back then. It was gated measurements, tracking generators and analyzers, "storage" scopes with persistent phosphor, and various other tricks.

Onwards - Don

Auto biasing is one of those things which in principle seems like a relatively easy problem. However, the devil is in those details, which makes it interesting.

 

[sergeauckland]

Auto biasing is one of those things which in principle seems like a relatively easy problem. However, the devil is in those details, which makes it interesting.

One issue with auto bias, which was recognised even in the 1950s, is that a class AB1 output valve can be biased either for continuous sine wave or speech and music duties. Distortion will be very different under the two conditions. The reason is that with speech and music (assuming reasonable dynamic range) the average current drawn by the valve is low, close to its quiescent bias level. With continuous tone, the current draw is much higher, and so the bias voltage will rise due to the bias being derived by the current through the cathode resistor. An amplifier designed for sine-wave duty will have the wrong bias (too low) on speech and music, and one designed for speech and music will have too much bias on tones. That makes it difficult to measure distortion accurately, although manufacturers who understood this at the time, would create a fixed bias so the amp could be measured. The cathode bypass capacitor created a long(ish) time constant that smoothed out the bias variations on speech and music, but that wouldn't work for tone which by its nature is continuous.

High power valve amps typically had fixed bias using a separate supply that didn't have the issue with bias shifting with signal, but this added considerably to cost so wasn't used on most lower powered amps.

S.