Double Blind tests *did* show amplifiers to sound different

[ferrellms]

You'd be surprised at how critically you have to listen for little noises and responses from the amplifier during situations like this. It literally takes hours with all sorts of program material to finally be able to sign off the design. There's no test equipment on Earth that can evaluate the subjective sound of leaves blowing in a soundtrack and how it relates to an amplifier not clipping those sounds as it goes to sleep. Things like the integration time of the circuit which listens to the audio and a lot of other details. And of course failing to catch these things would be extremely critical if a customer calls to complain about it!

I doubt this, and there is no real evidence for it other than your claim for the truth of it. I am an atheist, too!

 

[MakeMineVinyl]

I doubt this, and there is no real evidence for it other than your claim for the truth of it. I am an atheist, too!

Okay... Whatever.

 

[Wes]

audiophiles do not evaluated amplifiers (or other components) in the highly constrained environment of that Blind ABX test. Therefore it doesn't seem axiomatic or common sense to me that the Blind ABX test would reveal all sonic differences.
...

should we apply that speculation to drug testing? DBT of drugs takes place in a different env. too

 

[March Audio]

Well, maybe actually do some work with a real amplifier manufacturer, and see how far not listening at all gets you?

I measure because it tells me things that listening won't / can't. I have yet to come across an amp or dac that measures well and sounds bad. I keep asking people to give me an example of a bit of kit that does, but so far zero responses.

Sure I listen as part of a design check but it's more of a paranoia thing "just in case" than anything else.

Clicks, pops and weird noises at on / off can all be measured, but it is probably just quicker to listen for them as a test. I would still then measure them to understand the nature of the event.

 

[Feanor]

Measurements of Pass amps have shown that he tweaks them to sound a particular way and doesn't shoot for total transparency.
If that is so, then we would expect there to be a audible difference.

Yes, it's true. Pass does create his amps to sound a certain way, and the X150.5 happens to be the best amp by far of those I've owned in 50 years. Honestly though, Amir would probably find the performance of say, the X150.5 to be "not recommended" or a least far from SOTA.

Here are John Atkinson's measurements for Pass INT-150 integrated which is essentially the X150.5 with a passive front end ... https://www.stereophile.com/content/pass-labs-int-150-integrated-amplifier-measurements. Nevertheless those measurements are not so bad that Amir would classify the X150.5 as "broken", so by the standards around the this place I ought not to be able to hear any real difference, i.e. differences I hear are my imagination.

Atkinson a bit latter measured the Pass XA60.5 class A monoblocks. Their measurements were rather similar to the X150.5, not surprisingly since they are of similar design ... https://www.stereophile.com/content/pass-labs-xa605-monoblock-power-amplifier-measurements . Atkinson in the same items said in the subjective section of the review said ...
"Summing Up: I don't have much to say about the Pass Labs XA60.5 other than this: It is the best-sounding amplifier I have ever used.

Unlike Amir and a many people around hear, John Atkinson actually listens to amplifiers.

 

[March Audio]

.

Unlike Amir and a many people around hear, John Atkinson actually listens to amplifiers.

What makes you draw that conclusion?

Of course objectivists listen, they are just aware of the frailties of sighted and uncontrolled subjective opinions. They are just not driven/influenced by irrelevant biasing factors.

 

[MakeMineVinyl]

Sure I listen as part of a design check but it's more of a paranoia thing "just in case" than anything else.

That's essentially what I do, but I bang on them pretty hard to find things that aren't really measurable such as turn-on / turn-off noises as the power supply collapses (sometimes an amplifier will emit a short oscillation on turn-off if things aren't right), and if there are any unexpected noises when inputs are plugged in / unplugged. I have heard instances where things did not sound right and subsequent troubleshooting found something amiss with the DSP software. In the current amplifier I'm working with, there's a lot of things going on with certain parameters in real time, and it takes lots of listening to determine that that's being done unobtrusively.

 

[March Audio]

That's essentially what I do, but I bang on them pretty hard to find things that aren't really measurable such as turn-on / turn-off noises as the power supply collapses (sometimes an amplifier will emit a short oscillation on turn-off if things aren't right), and if there are any unexpected noises when inputs are plugged in / unplugged. I have heard instances where things did not sound right and subsequent troubleshooting found something amiss with the DSP software. In the current amplifier I'm working with, there's a lot of things going on with certain parameters in real time, and it takes lots of listening to determine that that's being done unobtrusively.

Sorry but all that it eminently measurable, I am a little shocked that you would say it isn't.

Scope, normal mode, capture.

 

[Sal1950]

As a real world example of things I have listened for on a current design, there is a sleep mode where in the absence of signal for 10 minutes, the amplifier goes into a low power mode. Yes, the threshold for "waking up" the amplifier is specified, but how does that actually sound when playing, say a real movie with extended soft dialogue? Does the amplifier clip off sound when it enters sleep? Does it clip off sound when it wakes up? Does the amplifier make strange noises or pops when it wakes up/goes to sleep? These things can only be determined by critical listening. If I just measured things, and shoved it out the door, I'd be toast.

You'd be surprised at how critically you have to listen for little noises and responses from the amplifier during situations like this. It literally takes hours with all sorts of program material to finally be able to sign off the design. There's no test equipment on Earth that can evaluate the subjective sound of leaves blowing in a soundtrack and how it relates to an amplifier not clipping those sounds as it goes to sleep. Things like the integration time of the circuit which listens to the audio and a lot of other details. And of course failing to catch these things would be extremely critical if a customer calls to complain about it!

I bought into your first post as being something good to check for post-production.
But then you went off on a tangent about how critically you have to listen for these things and how "no test gear on earth" could reveal any of it in the amps output signal.
I can just see it now, your customer with his ear pressed tightly to the speaker, patiently waiting to see if he can hear any extremely minute abnormality in the wavefront as the amp "goes into or wakes from sleep mode". LOL

But surely, it it measures perfectly, there is no point in any listening at all, no?

Yep, Probably should have stopped there. LOL

 

[MakeMineVinyl]

Sorry but all that it eminently measurable.

NO! Some things are just easier to root out by simply listening. If there's a transient noise that occurs imtermittently, that would lead to further investigation, but that wouldn't necessarily be caught by instrumentation unless you just happen to have a scope going when it happens.

I honestly don't comprehend the aversion to doing listening testing around here. Is there something 'yall are trying to prove that only wimps listen?

 

[DonH56]

I think all the folk here listen, frequently. The issue is when folk dismiss measurements in favor of sighted (biased) listening tests and attempt to draw far-reaching conclusions from them. A full set of measurements tells a lot (maybe not all, and of course I have not defined "full") and generally with much greater resolution than hearing. OTOH "all amplifiers sound the same" based on limited measurements and test cases is as weak as "all amplifiers sound different" based on uncontrolled hearing tests. Anytime extreme viewpoints are taken and the other side ignored results in threads like this.

 

[MakeMineVinyl]

I bought into your first post as being something good to check for post-production.
But then you went off on a tangent about how critically you have to listen for these things and how "no test gear on earth" could reveal any of it in the amps output signal.
I can just see it now, your customer with his ear pressed tightly to the speaker, patiently waiting to see if he can hear any extremely minute abnormality in the wavefront as the amp "goes into or wakes from sleep mode". LOL

Yep, Probably should have stopped there. LOL

Sorry, I do a lot of listening to something which is going to manufactured in the thousands, because I don't particularly want to have to explain the necessity of shipping back an 80 pound amplifier because of lack of due diligence on my part. And yes, what I'm designing now breaks a lot of new ground, so it damn well requires extensive listening to prove out techniques which haven't been used in the way I'm doing it. I'd elaborate further, but at this point it seems unlikely to sink in.

 

[March Audio]

NO! Some things are just easier to root out by simply listening. If there's a transient noise that occurs imtermittently, that would lead to further investigation, but that wouldn't necessarily be caught by instrumentation unless you just happen to have a scope going when it happens.

I honestly don't comprehend the aversion to doing listening testing around here. Is there something 'yall are trying to prove that only wimps listen?

Sorry thats nonsense and you are trying to move the goalposts. Why wouldnt you have a scope going if you are trying to find a problem? Anything you can hear, certainly of the nature you have described, is quite measurable. If you hear a problem you then crack out the instrumentation to measure and analyse it so that you can understand whats going on. Otherwise you are fishing in the dark.

There is no aversion to listening.

A specific example I have to hand. I failed an amp on production test a few weeks ago due to a click at switch off. Precisely the sort of issue you previously described. Scope easily revealed just how bad it was. Here we easily capture an event that was just 20uS in duration. Nothing difficult about it at all. Faulty module.

 

[Vasr]

I OTOH "all amplifiers sound the same" based on limited measurements and test cases is as weak as "all amplifiers sound different" based on uncontrolled hearing tests.

This.

I wish there was a middle ground to explore what perceived (and validated) audibility differences between amps could be correlated with in measurements.

Instead I keep hearing, we got every test we need and/or any perceived difference must be because of insufficiently rigorous hearing procedure. That kind of kills any reasonable investigation in that direction by setting up a barrier that is difficult to achieve.

 

[SIY]

@Feanor You keep confusing "listens" with "peeks."

I wish there was a middle ground to explore what perceived (and validated) audibility differences between amps could be correlated with in measurements.

This is well known. Frequency response (including variations from finite source impedances driving speakers), stability, noise, overload characteristics, power, and to some extent distortion. There is exactly zero evidence of any mysterious "we can hear it but can't measure it" phenomena.

 

[Vasr]

You keep confusing "listens" with "peeks.".

Where did I do that?

There is exactly zero evidence of any mysterious "we can hear it but can't measure it" phenomena.

I didn't say this either. I am outta here.

 

[SIY]

Where did I do that?

Sorry, editing error on my part. That part was supposed to be a reply to Feanor. I'll correct that.

 

[March Audio]

This.

I wish there was a middle ground to explore what perceived (and validated) audibility differences between amps could be correlated with in measurements.

Instead I keep hearing, we got every test we need and/or any perceived difference must be because of insufficiently rigorous hearing procedure. That kind of kills any reasonable investigation in that direction by setting up a barrier that is difficult to achieve.

Its not as difficult as you imagine, certainly with components like amps and dacs. Accurately matching volume levels and simply not being able to see the device(s) under test goes a long way to solving the problem. I have done this with many audiophiles and the differences they could hear when sighted and uncontrolled disappear or become very much reduced.

 

[Vasr]

Its not as difficult as you imagine, certainly with components like amps and dacs. Accurately matching volume levels and simply not being able to see the device(s) under test goes a long way to solving the problem. I have done this with many audiophiles and the differences they could hear when sighted and uncontrolled disappear or become very much reduced.

No skin in the game, but if you wanted to do this with scientific rigor to find relationship between audible differences and measurements (not whether you want to discredit some audiophiles or not which is not a very scientific goal):

1. Set up arranged by someone with no skin in the game to remove any implicit bias in the set up
2. Calibration of the set up and reference sample - Pick two amps with varying measurement characteristics and test to make sure that anyone can detect the extremes (otherwise the test is biased towards not hearing a difference). Now move to better and better measurements until the same people cannot differentiate with statistical significance.
2a. Drink beer.
3. Select sample test subjects with no bias towards age/sex/etc
4. To test at what difference in measurements the differences become inaudible in a DBT
4a. Use the calibration amps above and test them pairwise with one being the top measuring and the other varying. For each pair in random order, subjects just have identify the same amp as A or B in any pair with better than coin toss probability when you switch them between A and B.
4b. Find the threshold at which the median cannot distinguish any more
4c. Drink beer.
5. To test whether two amps sound the same or not (i.e., people can reliably hear a difference to pick one out). They don't have to pick which one they like.
5a. Select the reference pair above and pick some similar measuring amps above the differentiating threshold above in measurements but across multiple technologies and vendors. Mix the amps pairwise randomly.
5b. Repeat the test above to let people pick which one is A and which one is B when they are randomly switched and listened to blind.
5c. Check that people reliably picked the reference pair - which was picked for audible difference. If not go back to Step 2
5d. Drink beer.
6. Tabulate results for each pair. Find pairs which people picked to statistical significance (better than coin toss). If none other than the reference pair, thesis that people don't hear any difference above that threshold of measurement proved.
If they did differentiate two amps to statistical significance, investigate why they both measured the same and whether the difference is explained by some other measurements. If the latter, repeat the test above with the new measurement added.
7. Drink beer
8. Post your results to ASR.

If the above is too much work or impractical, avoid voicing definitive opinions on whether any set of measurements capture all differences or whether two amps sound different despite measurements

OR

Continue with the my opinions vs yours as usual with no scientific validity on either side.

 

[DonH56]

This.

I wish there was a middle ground to explore what perceived (and validated) audibility differences between amps could be correlated with in measurements.

Instead I keep hearing, we got every test we need and/or any perceived difference must be because of insufficiently rigorous hearing procedure. That kind of kills any reasonable investigation in that direction by setting up a barrier that is difficult to achieve.

Measurements are much more resolving and accurate than our ears. Decades of research proves that. IME measurements consistently back up hearing tests. The barrier is not that difficult, but requires more rigor than most 'philes want to put into it (which is usually "none" since they "know" they can hear it, and do not want their hearing beliefs questioned). The problem is blind faith in the infallibility of their hearing. I used to be that way, hearing every misaligned grain of copper in an interconnect, then many years ago an eye-opening (and very humbling) series of blind and double-blind tests showed me just how wrong I was about most all the differences I (thought I) heard. And revealed the cause for the ones I did, which I found interesting.

Setting up a rigorous test can take some effort, but for line-level stuff there are all sorts of ABX tests you can perform using your computer. And some basic controls like level matching and keeping the volume low enough to avoid clipping are not too hard to implement and will yield pretty good results much of the time for any system. I have set up such tests for friends, and helped run a few blind test with them, that often showed how the differences went away with a few basic controls. And how some did not.

For myself, I would not say "any perceived difference must be because of insufficiently rigorous hearing procedure." Rather I would say you need a decent listening test to confirm a difference, the first step in tracking down what is causing the difference. If I think I hear a difference, then make a few measurements and discover gain was not matched, that probably invalidates what I heard. If those same measurements show a difference in frequency response, a higher noise floor, or something like that, then I know why I heard a difference.

I think a fundamental problem is that people do not trust the designers or engineering; IME/IMO they have been steered and often misled by marketing to believe in things that either don't exist or do not apply. Many ploys use real science to show things that are inaudible and convince consumers they are deeply significant. Blow up the y-axis on a plot with no x-axis to show a huge roll-off in a cable, and do not mention that the actual droop is say 1 dB at 1 MHz, and the flatter cable looks much better in the brochure but the difference is meaningless in the real world. By that token I should be rebadging the 40 GHz RF cables we use in our lab and selling them as super wideband audio cables.

Engineers are not alone; there are people who do not trust doctors or most any professionals. And believers or schemers who prey on that fear to sell all sorts of magical goods that defy science. It does bug me when people who do not know the science, and do not care to try to learn or understand it, decide they don't trust it, and thus I (we) must simply be wrong and we should "agree to disagree". My analogy is along the line of "lots of science shows the sun is very hot, but since I have not been there, I cannot prove it to you, so all science is unproven".

I keep dredging up this old post I made many years ago in another place but I think it still represents the divide when things are taken to the absurd: https://www.audiosciencereview.com/forum/index.php?threads/the-good-forum.285/page-3#post-8454