• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Limitations of blind testing procedures

Status
Not open for further replies.

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
Carver matched transfer functions to make his SS amp sound like a tube amp. I wonder what the results would have been if he had been challenged to make the tube amp sound like the SS? Seems like a tougher task to me... IIRC one of the biggest features of the resulting SS amplifier product was a resistor to raise the output impedance of the SS amp, not sure what else. Long ago.
That's what I thought. It effectively says that amplifiers have a fixed transfer function, and a fixed output impedance. This is not the case - but it is what a lot of people imagine, perhaps.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
That's what I thought. It effectively says that amplifiers have a fixed transfer function, and a fixed output impedance.

Why do you think that it effectively says that?


This is not the case - but it is what a lot of people imagine, perhaps.

At least wrt to the output impedance (imo the concept/existence of the transfer function is rarely discussed/acknowledged) it is presumably true; the reason might be that often an output impedance number is given in datastheets instead of a graph showing the impedance over the frequency range.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
Why do you think that it effectively says that?
Because in claiming you can simulate "the transfer function" of one amplifier using a different type of amplifier and tacking on a few components, you have to be assuming that it is a simple, fixed transfer function.
 

SoundAndMotion

Active Member
Joined
Mar 23, 2016
Messages
144
Likes
111
Location
Germany
Because in claiming you can simulate "the transfer function" of one amplifier using a different type of amplifier and tacking on a few components, you have to be assuming that it is a simple, fixed transfer function.
I’m not sure what people here understand as “transfer function matching”, but if two amps have the same input/output relationship, they must have very similar transfer functions. Carver did not have access prior to the challenge to the transfer functions. He used “null difference testing”. See here.
I’m also not sure what you meant earlier by “mythology of the Carver Challenge”, but he really succeeded in the challenge twice, by really modifying real amps to match the sound of other amps and pass a blind listening test.
 

The Smokester

Active Member
Joined
May 11, 2017
Messages
136
Likes
39
Location
SF Bay
Because in claiming you can simulate "the transfer function" of one amplifier using a different type of amplifier and tacking on a few components, you have to be assuming that it is a simple, fixed transfer function.

With modern DSPs, seems that one could do this trick with any amplifier with sub-auditory linearity.

Are modern amplifiers not more and more linear? I haven't kept up, but I expect that the audible difference must be getting smaller.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I’m not sure what people here understand as “transfer function matching”, but if two amps have the same input/output relationship, they must have very similar transfer functions. Carver did not have access prior to the challenge to the transfer functions. He used “null difference testing”. See here.
I’m also not sure what you meant earlier by “mythology of the Carver Challenge”, but he really succeeded in the challenge twice, by really modifying real amps to match the sound of other amps and pass a blind listening test.
I do find the world of audiophilia very strange! We seem to have two ideas held in people's heads at the same time:
(1) Even changing a digital cable will result in changes that can be heard but are so tiny they cannot measured.
(2) A mere 70dB null (that even then seemed somewhat 'elusive' according to the account - I am not surprised) and auditioning amps with a re-wiring exercise for every changeover is proof that two amplifiers sound identical.

The craziest part of all, of course, is the idea that an amplifier should have a sound at all, and that price is a proxy for quality.

Edit: for (1) we could substitute the difference between 16 and 24 bits; the effect of minimum phase versus linear phase reconstruction filters; interconnects in general; audible differences between various brands of DAC with noise and distortion in the -115dB region. Basically all the things that even 'science'-oriented audiophiles think make an audible difference but are way, way below -70dB in difference.
 
Last edited:

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
<snip>
I’m also not sure what you meant earlier by “mythology of the Carver Challenge”, but he really succeeded in the challenge twice, by really modifying real amps to match the sound of other amps and pass a blind listening test.

Afair the people of the audio critic did finally a blind test (although, again afair, the starting point was a percepted difference between amplifiers in sighted listening tests) but the stereophile people did not.
 

The Smokester

Active Member
Joined
May 11, 2017
Messages
136
Likes
39
Location
SF Bay
...The craziest part of all, of course, is the idea that an amplifier should have a sound at all, and that price is a proxy for quality.

I think this is simply explained by noting what an individual is, consciously or not, trying to achieve:

If it's to reproduce a live performance as closely as possible, then one presumably wants an amplifier as linear as necessary.

If one wants a particular "sweet" sound, then maybe a tubed SET is preferred.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
With modern DSPs, seems that one could do this trick with any amplifier with sub-auditory linearity.

Are modern amplifiers not more and more linear? I haven't kept up, but I expect that the audible difference must be getting smaller.
Yes, I think it would be possible to simulate a multidimensional transfer function using DSP, but really you are creating an effects unit, not a better amp (regardless of how much it costs).

A primitive (and therefore expensive) amplifier's transfer function would not just be a simple Vin versus Vout 2D graph, but would have the element of frequency response variation (the output impedance I think you mentioned which will vary with signal content) plus various kinks that come and go with signal content and signal history, due to transformer hysteresis, power supply droop, thermal effects on the amp and load, and so on.
 

The Smokester

Active Member
Joined
May 11, 2017
Messages
136
Likes
39
Location
SF Bay
Yes, I think it would be possible to simulate a multidimensional transfer function using DSP, but really you are creating an effects unit, not a better amp (regardless of how much it costs).

A primitive (and therefore expensive) amplifier's transfer function would not just be a simple Vin versus Vout 2D graph, but would have the element of frequency response variation (the output impedance I think you mentioned which will vary with signal content) plus various kinks that come and go with signal content and signal history, due to transformer hysteresis, power supply droop, thermal effects on the amp and load, and so on.

Good points. I think these mean that excellent engineering still counts.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
Has everyone forgotten besides the more famous challenge with the Conrad Johnson amp Carver did the same thing with a Mark Levinson amp for a different publication? As I recall he did have to change out some electrolytic decoupling caps in his amp to polypropylene caps to manage this.

Carver's thinking was quite simple though it still is not believed by audiophiles. He knew he had built a powerful, fairly wide bandwidth, low distortion amplifier that should be quite high fidelity to the input signal. Any amplifier that sounded different than his had to have some coloration which meant a smaller performance envelope. The challenge was to color his amp to match. A challenge and requiring some skill and knowledge on his part, but in principle a lead pipe cinch.

Stereophile actually made it easy by picking a large tube amplifier. Restricted bandwidth, higher distortion smaller performance envelope. He did use nulling to check his work, but also distortion meters and a few other bits of gear. They mentioned he added hand-wound coils, some resistors in places and some output resistance. The final piece to make it match with a real speaker attached was to weaken the power supply in his amp as the C-J had a slightly uncontrolled upper bass due to PS limitations.

I've written before how I loaded down a triode amp at the output, tapped the output and fed a good clean SS amp, and the result sounded like the tube amp. The tube amps perceived superior qualities are in fact all colorations. Seems few people believe me either. People who love SET's should pick an inexpensive very low powered one they like and put it in front of a good SS amp. They could have their cake and eat it too. Sound character they like with less restrictions on which speaker to use. Yes I have also done this.

And yes you can do the same thing in DSP. It often is simpler instead of more complex than people believe. UAD has made a business of this on the pro side. You buy the plug in you want.
http://www.uaudio.com/uad-plugins/all-plugins.html

They take the real gear, and work out all the interactions even down to cross coupling among circuitry and transformers due to physical layout. You buy the plug in and have the sound. If audiophiles would quit being ridiculous half a** mystics they could benefit from this with much less expense. No one is going to invest time or effort in it however, because the goofy folks in this hobby would never agree it is possible. And that includes no matter how much you demonstrate to them it is so. Despite the old catch phrase, no one distrusts their ears as much as audiophiles.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
Has everyone forgotten besides the more famous challenge with the Conrad Johnson amp Carver did the same thing with a Mark Levinson amp for a different publication? As I recall he did have to change out some electrolytic decoupling caps in his amp to polypropylene caps to manage this.

We had a brief discussion about two of the challenges in the "meet Bob Carver Tomorrow" thread:
http://audiosciencereview.com/forum/index.php?threads/meeting-bob-carver-tomorrow.1255/page-3

But i´ve only read about the other challenges after the stereophile published the event and the letter by Harvey Rosenberg and had totally forgotten about it until you mentioned it in that thread.
The stereophile challenge version is a regular topic in forum discussions, while the audio critic is not, presumably due to the reason Rosenberg mentioned in his letter.

Carver's thinking was quite simple though it still is not believed by audiophiles. He knew he had built a powerful, fairly wide bandwidth, low distortion amplifier that should be quite high fidelity to the input signal. Any amplifier that sounded different than his had to have some coloration which meant a smaller performance envelope. The challenge was to color his amp to match. A challenge and requiring some skill and knowledge on his part, but in principle a lead pipe cinch.

Unfortunately i don´t know about the details of the (possibly) parts exchange, but if Carver really replaced an electrolytic by a polypropylen then chance are quite high that the assertion "the challenge was to color his amp to match" wasn´t correct.

I've written before how I loaded down a triode amp at the output, tapped the output and fed a good clean SS amp, and the result sounded like the tube amp.

Who could i know that is really sounded like the tube amp? Without a description of the controlled listening tests done to corrobate that impression it is impossible to evaluate.

They take the real gear, and work out all the interactions even down to cross coupling among circuitry and transformers due to physical layout. You buy the plug in and have the sound. If audiophiles would quit being ridiculous half a** mystics they could benefit from this with much less expense. No one is going to invest time or effort in it however, because the goofy folks in this hobby would never agree it is possible. And that includes no matter how much you demonstrate to them it is so. Despite the old catch phrase, no one distrusts their ears as much as audiophiles.

It´s really amazing that "audiophiles" don´t trust in these attempts despite all the ridicule and belitteling that routinely takes place.... ;)
But maybe it´s the inconsistency that is evident by using the Carver challenges as evidence. As said in the other thread/posts, if taken seriously, the audio critic people were convinced that perceptable differences existed between solid state amplifiers, because that was the starting point for the challenge (all according to the then published material). As long as there at the end a "hah those dumb audiophiles" possible to express, nobody seems to challenge the premise. What about the measured differences compared to the known thresholds of hearing?
At the end the audio critic people wrote to have done a "blind listening" and came to the conclusion that there wasn´t a perceptable difference (not anymore?), as usual the documentation of those "blind tests" was ....let´s say sparse.

In the stereophile challenge there was presumably a measureable difference partly above the know hearing thresholds, but the stereophile crew only did sighted listening (althoug prepared to do "blind" as well) and concluded that the amplifiers sounded the same after the modifications.

So what are the possible conclusions:
-) sighted listening works as a charm, first to detect the audible differences below the known hearing thresholds and at the end to detect the "sameness" , but why then all this blaming of subjective reviews, that stereophile and the like do?
-) there never were audible differences and everything else was much ado about nothing to present a good show and have a nice marketing story to tell at the end

-) the "nongolden ear brigade" doesn´t take their own arguments serious when presenting the challenge results
 

Purité Audio

Master Contributor
Industry Insider
Barrowmaster
Forum Donor
Joined
Feb 29, 2016
Messages
9,049
Likes
12,147
Location
London
It's really very simple, everyone likes to feel 'special' , believing that one can hear differences that only 'golden eared' reviewers can hear makes them feel special, these perceived differences disappear under unsighted comparisons because in reality they do not exist.
Keith
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
<snip>these perceived differences disappear under unsighted comparisons because in reality they do not exist.
Keith

Which is probably just an incorrect assertion. :)
It might be true, that in the usual misguided approach to testing (i.e. if it´s "blind" then it is already "good" ; level match etc. provided), the null hypothesis can´t be rejected (no significant result obtained), but whenever a test is better planned and executed - means there is a training phase and participants have time to get used to the protocol, sample size is bigger and so on) then there most likely is some evidence for the EUT.
 

Purité Audio

Master Contributor
Industry Insider
Barrowmaster
Forum Donor
Joined
Feb 29, 2016
Messages
9,049
Likes
12,147
Location
London
I am afraid Jakob you are just full of hot air, if I level match to .1dB and am unaware of which component I am listening to and I cannot hear a difference that is good enough for me, I would rather spend time pursuing real ,tangible improvements to sound quality.
Keith
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
If it has good measurements it will sound good ,the two go hand in hand, you may prefer something with more distortion of course.
Keith
The AES (PNW section) has a nice page with laws for audio engineers and von Recklinghausen is quoted with:

If it measures good and sounds bad, it is bad. If it Measures bad and sounds good, you've measured the wrong thing.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
I am afraid Jakob you are just full of hot air, if I level match to .1dB and am unaware of which component I am listening to and I cannot hear a difference that is good enough for me, I would rather spend time pursuing real ,tangible improvements to sound quality.
Keith
First, please reread my post which includes a citation of your post that i responded too. Do you miss the "Is" in your post too? :)
Your personal belief is just that and i´m fine with it (though the implicit assertion that your strong bias surely can´t have an impact is ....err, umh....questionable), but your post contained a categorical assertion about disappearing differences in general, hence my comment.
 

Purité Audio

Master Contributor
Industry Insider
Barrowmaster
Forum Donor
Joined
Feb 29, 2016
Messages
9,049
Likes
12,147
Location
London
Really this is too dull to continue, but before I check out, I can hear genuine differences unsighted and level matched if differences exist , if I cannot hear a difference between two components level matched and unsighted then that is good enough for me as far as I am concerned there is no difference.
I really am only interested in improvements in audio quality , measureable improvements, for example look at the frequency response plots of the Dutch &Dutch 8Cs and Kii THREEs I posted , even slight differences are inconsequential , large quantifiable improvements that is what interests me.
Keith
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
Really this is too dull to continue, but before I check out, I can hear genuine differences unsighted and level matched if differences exist , if I cannot hear a difference between two components level matched and unsighted then that is good enough for me as far as I am concerned there is no difference.
I really am only interested in improvements in audio quality , measureable improvements, for example look at the frequency response plots of the Dutch &Dutch 8Cs and Kii THREEs I posted , even slight differences are inconsequential , large quantifiable improvements that is what interests me.
Keith
Which is all fine, but had really nothing in common with your post that i cited and commented to:

It's really very simple, everyone likes to feel 'special' , believing that one can hear differences that only 'golden eared' reviewers can hear makes them feel special, these perceived differences disappear under unsighted comparisons because in reality they do not exist.
Keith
There you were clearly talking about others.....

Btw, another favourite one from the AES page is:

"There is always a simple answer to every problem, neat plausible and wrong" (Harrison´s Law)

( http://www.aes.org/sections/pnw/laws.htm )
 
Status
Not open for further replies.
Top Bottom