Limitations of blind testing procedures

[amirm]

The AES (PNW section) has a nice page with laws for audio engineers and von Recklinghausen is quoted with: If it measures good and sounds bad, it is bad. If it Measures bad and sounds good, you've measured the wrong thing.

Careful! First, that is a bit of fish story since I have never found the source of that line. It is attributed to a meeting at Boston Audio Society yet I have never found it. So the context is totally unknown.

What is not unknown is that the late Recklinghausen was the key driver behind the IHF, the Institute of High Fidelity. IHF set standards for fidelity of amplifiers using measurements. Here is Recklinghausen talking about said standard: http://www.bassboy.com.au/getreel/site/classicamps/files/articles/ihf/article.htm

"The new Standard will help establish design goals for audio engineers and at the same time furnish test techniques for validating them. For the audiophile, the new ratings will make possible a more intelligent choice among the profusion of amplifiers now available."

He was also the editor of Journal Of AES for many years when papers were published showing the fallacy of sighted tests. As a member of review board/editor, he helped form and shape the current accepted standard of throwing you out of the room if you show up with sighted tests as your proof for anything.

Finally, he was the chief engineer at H.H. Scott. In that world, folks did try to win the race with better specs so it is not out of reason that he would try to deflect that superiority by others in that regard.

Speaking of that, I ran into your posts on a German forum where you had signed them as seller/manufacturer of audio amplifiers. Is that so?

Back to that quote, if you indeed find that the true sound of something is better despite worse measurements, indeed those measurements are faulty. The problem we deal with in this topic is that imagined sound is substituted for what is really output by the equipment. Recklinghausen would never get behind that notion.

 

[Cosmik]

...if you indeed find that the true sound of something is better despite worse measurements, indeed those measurements are faulty.

I thought the mantra was that we have preference, but that doesn't mean "better"..? "Better" would be a subjective judgement only, and therefore the above sentence can never actually be true..?

 

[DonH56]

That's what I thought. It effectively says that amplifiers have a fixed transfer function, and a fixed output impedance. This is not the case - but it is what a lot of people imagine, perhaps.

A transfer function can be a very complicated thing... I am not sure what you mean by "fixed", but engineers know that nice LTI (linear time-invariant) transfer functions that are independent of the input signal and load exist only in textbooks and beginning exercises... I am pretty durn sure Bob Carver knew and knows how a transfer function can vary and that the output impedance is part of the transfer function. Reducing a myriad of complex interrelated variables to the two words "transfer function" is technically correct but one must understand what is behind those two little words.

FWIWFM - Don

 

[Ron Party]

I thought the mantra was that we have preference, but that doesn't mean "better"..? "Better" would be a subjective judgement only, and therefore the above sentence can never actually be true..?

Exactly. The "if it Measures bad and sounds good, you've measured the wrong thing" is, sometimes, the scientific method denier's get out of jail free card. As JJ has stated, preference is inviolate, but it does not extend beyond the tip of one's nose. The room/gear itself is not necessarily the thing which has either not been measured or not been satisfactorily measured. It very well may be the listener who has not been measured or satisfactorily so. The listener simply may be unqualified to render an opinion as to what is *better".

 

[Jakob1863]

<snip>
What is not unknown is that the late Recklinghausen was the key driver behind the IHF, the Institute of High Fidelity. IHF set standards for fidelity of amplifiers using measurements. Here is Recklinghausen talking about said standard: http://www.bassboy.com.au/getreel/site/classicamps/files/articles/ihf/article.htm

Somehow i got the impression that you think his efforts at the IHF contradicts the quote, am i mistaken?
I´m wondering, because i´d say it follows the same line of thinking, see his comment within the linked article, where he wrote about the unhappy situation that two amplifiers could have quite similar measurements but very different sound quality.

And the quotation (even if it was not correctly attributed to him) reflects the position that the human listener is still the final arbiter wrt sound quality. Which imo makes sense as long as we aren´t able to reproduce original soundfields.

Speaking of that, I ran into your posts on a German forum where you had signed them as seller/manufacturer of audio amplifiers. Is that so?

Signed as developer, manufacturer and at the end distributor of that stuff; today i´m mainly working in consulting, development of audio gear and other electronics as well.
Does it matter?

Back to that quote, if you indeed find that the true sound of something is better despite worse measurements, indeed those measurements are faulty.

But that´s not his quote; he was not talking about worse measurements, but about good measurements nevertheless associated with bad sound quality.

The problem we deal with in this topic is that imagined sound is substituted for what is really output by the equipment. Recklinghausen would never get behind that notion.

Although i think i understand what you mean, basically any stereo two channel system relies on the fact that listeners are able to substitute a lot of imagined for what is really output by the equipment.

In fact i thought we were mainly discussing limitations of test procedures and imagined differences (wrt to test protocols and audio gear).

 

[Cosmik]

...any stereo two channel system relies on the fact that listeners are able to substitute a lot of imagined for what is really output by the equipment.

I see that sort of sentiment quite a lot, or "Stereo is an illusion". I don't believe it. It may be that the human goes into a different 'mode' of listening, but I don't think they are filling in any information that isn't there. When watching a black and white film, we don't imagine colour - we just adapt to the lack of colour.

 

[Frank Dernie]

" If it Measures bad and sounds good, you've measured the wrong thing"
Many bits of expensive (and inexpensive) hifi which measure poorly are very highly regarded by many enthusiasts.
It can be almost guaranteed that certain of the Stereophile writers will have waxed lyrical about products which John Atkinson's measurements show to be technically poor.
I remember Martin Colloms reviewing very, very expensive Audio Note kit and praising the sound quality and being surprised by how badly they measured.
He asked the hypothetical question "do they sound good because of the poor measurements or despite them".
I am of the firm opinion it is because of them - some people prefer the distortion they add or the modified frequency response. IMO there is no other plausible explanation which accounts for both the auditioning opinion and measurement results.

I have been totally exasperated by the proponents of SET amps praising the sound and saying it is audibly superior whilst asserting that the high levels of distortion are inaudible. You couldn't make it up.

 

[Jakob1863]

I see that sort of sentiment quite a lot, or "Stereo is an illusion". I don't believe it. It may be that the human goes into a different 'mode' of listening, but I don't think they are filling in any information that isn't there. When watching a black and white film, we don't imagine colour - we just adapt to the lack of colour.

Probably it depends on the meaning of "illusion". At a first glance it is related to the virtual sound sources that we perceive they aren´t there (at least they aren´t there, where we think they are), further it is related to the depth illusion of the sound field. As already pointed out in the articles from the 1930s, the developers noticed that the ability to perceive the "depth of image" was very different among listeners.

Imo to experience a certain degree of realism with a reproduction system, that is as far away from reality as two channel stereophonie is, there has to be a lot of fill in from listeners; don´t get me wrong it works quite good most of the time...

Back to the virtual sound sources; at that point i´d admit that the word "illusion" might be misleading as most of us can´t avoid the illusion, no matter if we see the loudspeakers or not, no matter how hard we try, we can´t perceive (proper setup provided) two discrete channels.
There is a small percentage of people who (afair for unknown reasons) aren´t able to perceive the virtual sound source and therefore quite often better stick to mono instead of stereo.

 

[Fitzcaraldo215]

" If it Measures bad and sounds good, you've measured the wrong thing"
Many bits of expensive (and inexpensive) hifi which measure poorly are very highly regarded by many enthusiasts.
It can be almost guaranteed that certain of the Stereophile writers will have waxed lyrical about products which John Atkinson's measurements show to be technically poor.
I remember Martin Colloms reviewing very, very expensive Audio Note kit and praising the sound quality and being surprised by how badly they measured.
He asked the hypothetical question "do they sound good because of the poor measurements or despite them".
I am of the firm opinion it is because of them - some people prefer the distortion they add or the modified frequency response. IMO there is no other plausible explanation which accounts for both the auditioning opinion and measurement results.

I have been totally exasperated by the proponents of SET amps praising the sound and saying it is audibly superior whilst asserting that the high levels of distortion are inaudible. You couldn't make it up.

It was a few years ago that there was a minor flap at Stereophile. MF's review heaped high praise on a Playback Designs player/DAC, yet JA's measurements revealed persistent low level noise, that was atypical of most digital playback gear. JA was not afraid to say that some folks just like a little added noise with their sound. Of course, Mickey would be attuned to that idea. Afterwards, Mickey and I had a little skirmish about that and other stuff at Hi Rez Hiway.

Playback Designs just ignored JA's measured issues in their Manufacturer's Comments, and it all blew over. But, I came away with somewhat more respect for JA's measurements and his comments after that episode.

 

[The Smokester]

I see that sort of sentiment quite a lot, or "Stereo is an illusion". I don't believe it. It may be that the human goes into a different 'mode' of listening, but I don't think they are filling in any information that isn't there. When watching a black and white film, we don't imagine colour - we just adapt to the lack of colour.

I think there is a large "sighted" component to a stereo image from speakers. Right now, I am confined to headphones and earphones for my serious listening. These are TOTL with Focal Utopias and JHarvey JH13v2 Freq custom IEMs. The sound is wonderful, as good as I've ever heard, but the stereo imaging can be comical. The spatial reference is your two ears so there is really no external fixed reference when your head moves. With no external reference point, your mind is freer to create surreal images. For instance, if an oboe starts you create an nice image near the center. Then a violin comes in, but it is not far off and you accommodate its position nicely and a nice "image" starts to form. Then the entire orchestra opens up with deep, concert hall bass which demands a large image. Your brain now can make choices like: Expand the image to beyond your ears (the reviewers' choice), or Expand your head. I've noticed it sometimes, if you concentrate too much, makes the weird choice.

 

[amirm]

Signed as developer, manufacturer and at the end distributor of that stuff; today i´m mainly working in consulting, development of audio gear and other electronics as well.
Does it matter?

It does. You have to show transparency in conversations and also show that your opinion is not polluted in the interest of your commercial enterprise. Best way to do that is to present independent research to back what you say. In the course of all of this, there should have been occasions for you to disclose this type of interest.

 

[amirm]

He asked the hypothetical question "do they sound good because of the poor measurements or despite them".
I am of the firm opinion it is because of them - some people prefer the distortion they add or the modified frequency response. IMO there is no other plausible explanation which accounts for both the auditioning opinion and measurement results.

There is one other plausible explanation which once I mention, you will agree. And that is, imagined improvements in sound. Since we don't know if they are really hearing anything new or different, then is impossible to know if it is due to technical reasons/measurements or not.

 

[amirm]

Somehow i got the impression that you think his efforts at the IHF contradicts the quote, am i mistaken?

No, his career and position in the industry are solid. He is a man of science and considers measurements critical in his work and professional collaborations. He is an AES Fellow, ex-president of AES, and editor of Journal of AES for many years.

So what is wrong is not with him. It is with people such who steal that one line from him with no proper attribution that he even said that ever and in what context as to distort his views and position him as an expert witness for sighted subjective evaluation of audio trumping measurements. It shows that the people doing that don't even know the man. The one liner was all that they needed to mis-position someone of such distinguished career.

In a less informed place you might get away with such a stunt but please don't do that here. Expect your statements to be analyzed deeply with much more context and knowledge than you see in typical audiophile forums where folks may accept these one-liners as gospel.

 

[amirm]

But that´s not his quote; he was not talking about worse measurements, but about good measurements nevertheless associated with bad sound quality.

What talk? Give me a link to an article or paper he wrote saying that.

 

[Cosmik]

No, his career and position in the industry are solid. He is a man of science and considers measurements critical in his work and professional collaborations. He is an AES Fellow, ex-president of AES, and editor of Journal of AES for many years.

So what is wrong is not with him. It is with people such who steal that one line from him with no proper attribution that he even said that ever and in what context as to distort his views and position him as an expert witness for sighted subjective evaluation of audio trumping measurements. It shows that the people doing that don't even know the man. The one liner was all that they needed to mis-position someone of such distinguished career.

In a less informed place you might get away with such a stunt but please don't do that here. Expect your statements to be analyzed deeply with much more context and knowledge than you see in typical audiophile forums where folks may accept these one-liners as gospel.

I am bemused by this whole conversation. Have I got it wrong, or is this quote one of many in a humorous list with an ironic title? People are talking as though it it is serious and important because it is on the AES web site and has the word "laws" in the title!

 

[amirm]

I am bemused by this whole conversation. Have I got it wrong, or is this quote one of many in a humorous list with an ironic title? People are talking as though it it is serious and important because it is on the AES web site and has the word "laws" in the title!

It is one in the same. It is a humorous set of quotes. It is the local chapter which is free to post fun things and is not meant at all to be taken as any official position of AES as you correctly guess.

Here is the original attribution by the way that I could find: http://hhscott.com/vonrecklinghausen.htm

I could find none of the references they mention to Boston Audio Society. Or Absolute Sound (which simply copies the pages from HH Scott per above).

I searched entire AES library and he has never taken such a position. Instead, he is the editor of the Journal where all the peer-reviewed papers on importance of blind testing was posted.

 

[Frank Dernie]

It was a few years ago that there was a minor flap at Stereophile. MF's review heaped high praise on a Playback Designs player/DAC, yet JA's measurements revealed persistent low level noise, that was atypical of most digital playback gear. JA was not afraid to say that some folks just like a little added noise with their sound. Of course, Mickey would be attuned to that idea. Afterwards, Mickey and I had a little skirmish about that and other stuff at Hi Rez Hiway.

Playback Designs just ignored JA's measured issues in their Manufacturer's Comments, and it all blew over. But, I came away with somewhat more respect for JA's measurements and his comments after that episode.

Similar this month, MF liked a ridiculously pricey streamer which had poor performance.
I am a fan of JA, he doesn't get involved in extended discussion but is pretty steadfast in his logic.

 

[Purité Audio]

Yes the Brinkmann 'Nyquist' , I too enjoy the disparity between the subjective and objective reviews.
Keith

 

[Cosmik]

Here's a good one:
https://www.stereophile.com/cables/805harm/index.html#DtvivklOsr3MRTwp.97

 

[The Smokester]

Here's a good one:
https://www.stereophile.com/cables/805harm/index.html#DtvivklOsr3MRTwp.97

I remember this review. Okay. Maybe it does sound better to Fremmer. But it can scarcely be called "reproduction" and it doesn't meet the requirement most of us have for our systems.