Master Thread: Are measurements Everything or Nothing?

[BDWoody]

This is the crux of my question, which measurements correlate with what someone might hear.

This is why I asked you earlier how well you understand them and what they mean. It isn't about the numbers in isolation to context, but how the measurements correlate to what may or may not be audible, and if so how.

Here are a couple of threads that might put a few pieces together for some following along who might want to try to understand.

This one is definitely worth a read.

Audibility thresholds of amp and DAC measurements

Introduction A recurring theme on ASR is whether or not the various measured qualities of the devices are audible. In this post, I'll present some clear and visual thresholds for when those imperfections can be considered a potentially audible concern. I will not explain the basics of amp/DAC...

audiosciencereview.com

This one gives a resource to test your own ears.

Distortion Listening Test

Ran into this interesting listening test by Klippel with regards to how much distortion you can hear. This is the link: http://www.klippel.de/listeningtest/lt/ Just casually wanting to see the test, I went with the defaults and for some reason, the levels were exceptionally low. I had to set...

audiosciencereview.com

And, this video by our host will help anyone who wants to compare components in a way that isn't meaningless, as they typically are without basic controls unless there are gross issues being dealt with.

 

[Blumlein 88]

This is the best post I have seen on the topic-


Not objectivity per se but naïve objectivity such as:

We can measure an impossibly high sinad so that must be best but with absolutely no correlation whatsoever to how it sounds to real people that have some experience in listening. This is the crux of my question, which measurements correlate with what someone might hear. I have bought amps previously which claim impossibly low distortion or sinad or whatever and they are trounced (in my view) by others that have less impressive numbers.

I dont have any issue with measurements and coming from a research background have done plenty of measuring myself but measurements which are not correlated with what the component is meant to do Ie "sound good" have little value in the pursuit of something which sounds "good".

I see lots of talk about measurements indicating which component is accurate. My experience thus far has been that some of these "accurate" devices sound pretty unlistenable. How is that accurate? ie the measurements do not represent accuracy at the ear.

What is the point of comparing devices on these parameters if we cannot show that there is a correlation with how something might sound.

I think a good parallel is wine, no measurement/s are good predictor of final quality as judged by an experienced taster/s

I then have to conclude that in terms of helping me decide which component is to my preference I cant trust measurements!

​

Others have commented usefully, but were your listening comparisons controlled at all or just hook it up and listen for a bit, and then switch. Trust us, we know how real sound differences are with listening done that way. Such listening methodology is in fact almost useless. You are influenced by so many things that aren't part of the sound.

Measurements within reason track with how things sound when you don't know what the source of the sound is. If you are listening sighted, you'll get results all over the place.

 

[jtgofish]

Others have commented usefully, but were your listening comparisons controlled at all or just hook it up and listen for a bit, and then switch. Trust us, we know how real sound differences are with listening done that way. Such listening methodology is in fact almost useless. You are influenced by so many things that aren't part of the sound.

Measurements within reason track with how things sound when you don't know what the source of the sound is. If you are listening sighted, you'll get results all over the place.

I subscribed to Hi Fi Choice magazine during a period of years in which they performed unsighted group reviews of products using a panel of experienced staff .These tests were performed in a dedicated room.All products were also professionally measured and those results were published together with the listening ratings and impressions.Significant differences and preferences were observed in all classes of components and there seemed to be very weak correlation between how things mesaured and the listening panel rankings.Typically in each group there were products which measured well and which sounded good but also ones that did not measure so well but also sounded good.
I believe an important factor in these tests was that the room was familiar to the listener panels which probaby helped in identifying sonic differences.Another was that the listeners were highly experienced people from within the industry.
I still use one component that was a group test winner from one of those tests and it still sounds excellent and has seen off many modern alternatives.

 

[Blumlein 88]

I subscribed to Hi Fi Choice magazine during a period of years in which they performed unsighted group reviews of products using a panel of experienced staff .These tests were performed in a dedicated room.All products were also professionally measured and those results were published together with the listening ratings and impressions.Significant differences and preferences were observed in all classes of components and there seemed to be very weak correlation between how things mesaured and the listening panel rankings.Typically in each group there were products which measured well and which sounded good but also ones that did not measure so well but also sounded good.
I believe an important factor in these tests was that the room was familiar to the listener panels which probaby helped in identifying sonic differences.Another was that the listeners were highly experienced people from within the industry.
I still use one component that was a group test winner from one of those tests and it still sounds excellent and has seen off many modern alternatives.

Are those available on the web anywhere? Would especially be interested in any they did not like which measured well.

Found a group test of interconnect. Hard to take them seriously. They say a blind group panel, but I now wonder about the particulars of how they do the listening. That is because there is no difference in the interconnects unless grossly mis-designed. My guess being they listened to several unlabeled wires, but knew it was A, B, C, D etc. In that condition it isn't hard for a group to convince themselves of one vs the other when in fact there is no difference. A good test would be to listen to two wires switched at random to see if they can spot when a difference occurs and when it is the same.

 

[GXAlan]

I have bought amps previously which claim impossibly low distortion or sinad or whatever and they are trounced (in my view) by others that have less impressive numbers.

My experience thus far has been that some of these "accurate" devices sound pretty unlistenable. How is that accurate? ie the measurements do not represent accuracy at the ear.

Accurate and preference are two different things. You a preferring a less accurate sound. That is OK. But it is not accuracy at the ear that you are favoring.

Take a beauty pageant. Do you want absolute accuracy with zero makeup? Or is some makeup better? What about the talent portion? Absolute accuracy? Or smoke and mirrors?

In the first, it’s easy to understand how preference and coloration can enhance underlying audio features. It can conceal blemishes. But it is inaccurate.

Measurements are able to tell you how much makeup has been put on. Tasteful? Or clownish/gaudy?

—-

1) La La Land Soundtrack
Emma Stone is an actress not a singer. She’s a very good singer, but she is clearly a better actress. Same with Ryan Gosling.

Take the track #9 or #12. Play it back on a Benchmark AHB2/LA90 and and a clean speaker you will get complete transparency. You are hearing an actor and actress. Their voice should feel as if it’s coming from something s little wider than 1 inch wide. That’s the magic of transparency. The person is really there. They’re not great singers when compared to someone who’s primary talent is singing.

Now play it back on a premium tube system from McIntosh, or even Ken Ishiwata Marantz solid state product. You will notice two things. One, the voice is larger than life. It’s coming from a phantom center image which isn’t a little over an inch wide but from an area in the center. Like auto tune there is something musical about it which is great. But the fact that it sounds like there’s a giant in front of you tells you that it’s not accurate.

For this soundtrack, hands down, a little bit of makeup to conceal blemishes and enhance definition and color is preferable.

2) Next try some well recorded classical music

I like Stewart Goodyear’s version of The Nutcracker Suite since it is a piece we are familiar with, played in a new way.

Tchaikovsky - The Nutcracker - Stewart Goodyear - Steinway & Sons

For his third Steinway & Sons label release, Stewart Goodyear returns to the music of Tchaikovsky with a world premiere recording, his own arrangment for solo piano of the complete Nutcracker ballet.

www.steinway.com

A Steinway has intrinsic harmonics that give it a richer tone than a Shigeru Kawai for example. Here, I think the same makeup that helped the first two tracks actually detracts. The better the transparency the better the sound.

Or even the jazz track from La La Land Soundtrack, Track #5. Again, the more transparent and better measuring setups will sound better.

 

[Blumlein 88]

This is the second time you've mentioned the Hi-Fi Choice reviews. The general opinion is that they're meaningless. Please allow me to explain why I (among others) have that opinion.

You wrote, " .... Significant differences and preferences were observed in all classes of components." In the first place, here is no mention of level-matching. If levels are not precisely matched, then the louder speaker will seem preferable. And you did mention "preferences", which is not the same as "more accurate". Preference is referenced to one person's personal likes or dislikes, and accuracy is referenced to reality as much as possible. As such, it's removed from the realm of "personal".

You used the phrase, " ... group reviews of products using a panel of experienced staff ... ". The way you structured this sentence leads me to believe that there were several listeners as well as several devices under review, all at the same time and in the same place. Multiple devices arranged in a row, can easily be identified by relative position. So the word "unsighted" might be misleading. Not only that, but a group of listeners, unless highly disciplined, will give subtle body-English cues in their reactions. Other listeners can pick up on these cues. If equipment is auditioned by a single person, alone in the room, then that disappears.

You also wrote, " ... Significant differences and preferences were observed in all classes of components and there seemed to be very weak correlation between how things measured and the listening panel rankings. " There are many pieces of equipment on the market, speakers, amps and interconnects inclusive, that have been designed and manufactured in such a way as to be euphonic, or to sound "pleasant" or "different" rather than accurate. If a device is designed to be accurate, it has only one point of comparison. But if a device is designed to be "pleasant", there are myriad upon myriad different results that can be obtained, each one of them sounding a little different. A listening panel should have no trouble determining which speaker or amp is neutral and accurate. But if they were auditioning equipment that was designed to be "euphonic", then each personal idea of "pleasant" would compel listeners to rate each device differently. Naturally, there would be little correlation between measurements and opinions, because personal likes and dislikes, which is what opinions are, would predominate.

Not only that, but a person infatuated with "euphonic" equipment may very well find accuracy to be distasteful.

One other thing you wrote was, " ... the listeners were highly experienced people from within the industry." To me, and probably to other people here, this means nothing. There are many con men and bunco artists who are experienced. That doesn't mean that we can trust them. The only description that means something is "disciplined and unbiased". THAT means something. There is nowhere any indication that this was the case for these "group reviews."

For all we know, they may have been drinking coffee and joking around with each other, making comments and comparisons during these auditions. We just don't know. And because we don't know, I don't give any credence to reviews like that.

Jim

From what little I can find, the listening is carefully level matched. It does appear the whole panel listens together. They make notes on the sound. Don't know if they talk during this or not. Don't know if they get more than one presentation or not. They do know when switches are done though not what gear they are listening to. So they expect differences. A check on reliability would be if they "switched" between 4 devices under review, and in fact only heard one. Does the listening panel then describe differences in their notes and have preferences? Every reasonable expectation is they will. If they did, it completely invalidates the entire testing procedure.

 

[jtgofish]

This is the second time you've mentioned the Hi-Fi Choice reviews. The general opinion is that they're meaningless. Please allow me to explain why I (among others) have that opinion.

You wrote, " .... Significant differences and preferences were observed in all classes of components." In the first place, here is no mention of level-matching. If levels are not precisely matched, then the louder speaker will seem preferable. And you did mention "preferences", which is not the same as "more accurate". Preference is referenced to one person's personal likes or dislikes, and accuracy is referenced to reality as much as possible. As such, it's removed from the realm of "personal".

You used the phrase, " ... group reviews of products using a panel of experienced staff ... ". The way you structured this sentence leads me to believe that there were several listeners as well as several devices under review, all at the same time and in the same place. Multiple devices arranged in a row, can easily be identified by relative position. So the word "unsighted" might be misleading. Not only that, but a group of listeners, unless highly disciplined, will give subtle body-English cues in their reactions. Other listeners can pick up on these cues. If equipment is auditioned by a single person, alone in the room, then that disappears.

You also wrote, " ... Significant differences and preferences were observed in all classes of components and there seemed to be very weak correlation between how things measured and the listening panel rankings. " There are many pieces of equipment on the market, speakers, amps and interconnects inclusive, that have been designed and manufactured in such a way as to be euphonic, or to sound "pleasant" or "different" rather than accurate. If a device is designed to be accurate, it has only one point of comparison. But if a device is designed to be "pleasant", there are myriad upon myriad different results that can be obtained, each one of them sounding a little different. A listening panel should have no trouble determining which speaker or amp is neutral and accurate. But if they were auditioning equipment that was designed to be "euphonic", then each personal idea of "pleasant" would compel listeners to rate each device differently. Naturally, there would be little correlation between measurements and opinions, because personal likes and dislikes, which is what opinions are, would predominate.

Not only that, but a person infatuated with "euphonic" equipment may very well find accuracy to be distasteful.

One other thing you wrote was, " ... the listeners were highly experienced people from within the industry." To me, and probably to other people here, this means nothing. There are many con men and bunco artists who are experienced. That doesn't mean that we can trust them. The only description that means something is "disciplined and unbiased". THAT means something. There is nowhere any indication that this was the case for these "group reviews."

For all we know, they may have been drinking coffee and joking around with each other, making comments and comparisons during these auditions. We just don't know. And because we don't know, I don't give any credence to reviews like that.

Jim

This is the second time you've mentioned the Hi-Fi Choice reviews. The general opinion is that they're meaningless. Please allow me to explain why I (among others) have that opinion.

You wrote, " .... Significant differences and preferences were observed in all classes of components." In the first place, here is no mention of level-matching. If levels are not precisely matched, then the louder speaker will seem preferable. And you did mention "preferences", which is not the same as "more accurate". Preference is referenced to one person's personal likes or dislikes, and accuracy is referenced to reality as much as possible. As such, it's removed from the realm of "personal".

You used the phrase, " ... group reviews of products using a panel of experienced staff ... ". The way you structured this sentence leads me to believe that there were several listeners as well as several devices under review, all at the same time and in the same place. Multiple devices arranged in a row, can easily be identified by relative position. So the word "unsighted" might be misleading. Not only that, but a group of listeners, unless highly disciplined, will give subtle body-English cues in their reactions. Other listeners can pick up on these cues. If equipment is auditioned by a single person, alone in the room, then that disappears.


One other thing you wrote was, " ... the listeners were highly experienced people from within the industry." To me, and probably to other people here, this means nothing. There are many con men and bunco artists who are experienced. That doesn't mean that we can trust them. The only description that means something is "disciplined and unbiased". THAT means something. There is nowhere any indication that this was the case for these "group reviews."

For all we know, they may have been drinking coffee and joking around with each other, making comments and comparisons during these auditions. We just don't know. And because we don't know, I don't give any credence to reviews like that.

Jim

I am sure the listeners in the group test that I still have would appreciate your thoughts on their professionalism and suitability for the job.
They included-
Andy Whittle-Head of Design at Rogers U.K.Now back at Rogers
Andy Sargeant-Designer at Audio Innovations and now owner of Pure Sound
Alan Sircom-journalist contributor at Hi Fi Plus
Paul Messenger-Editor/contributor at Hi Fi Critic which is arguably the most credible audio publication and is a no advertising subscription only publication.

 

[Deleted member 21219]

I am sure the listeners in the group test that I still have would appreciate your thoughts on their suitability for the job.
They included-
Andy Whittle-Head of Design at Rogers U.K.Now back at Rogers
Andy Sargeant-Designer at Audio Innovations and now owner of Pure Sound
Alan Sircom-journalist contributor at Hi Fi Plus
Paul Messenger-Editor/contributor at Hi Fi Critic which is arguably the most credible audio publication and is a no advertising subscription only publication.

Your reply is classified as "Appeal To Authority". It's one of the many common rhetorical fallacies.
As for "being sure that they would appreciate my thoughts" .... it would be very nice to have them post here to the ASR forum. We could perhaps get their views on the points that were raised.

Jim

 

[Blumlein 88]

I am sure the listeners in the group test that I still have would appreciate your thoughts on their professionalism and suitability for the job.
They included-
Andy Whittle-Head of Design at Rogers U.K.Now back at Rogers
Andy Sargeant-Designer at Audio Innovations and now owner of Pure Sound
Alan Sircom-journalist contributor at Hi Fi Plus
Paul Messenger-Editor/contributor at Hi Fi Critic which is arguably the most credible audio publication and is a no advertising subscription only publication.

Unless they are not human, none of their creds exempts them from coming to unwarranted conclusions in such testing as it appears.

I'd ask, is there anywhere that describes in full detail how they do their testing?

Also, suppose this is how it is done:
Without knowing which components are which they are given 4 things to compare. They listen for 30 minutes to each while taking notes. They all listen together at the same time. They know when a switch is made between each of the components.

Do you think this is adequate for comparing things or not?

 

[solderdude]

I then have to conclude that in terms of helping me decide which component is to my preference I cant trust measurements!

That is totally correct and when you feel that way then you should decide which components to buy based on your preference.
Not based on measurements nor based on recommendations.
When you fully trust the 'highly experienced authorities' that recommend you things using flowery wording then use that as a guide line.

That simply means that you have a preference (like everyone has) and that is not necessarily 'high signal quality' but rather a certain coloration that suits your preference.

This is the reason why YOU cannot correlate specific numbers created by measurement gear to your personal experience.
That does not mean there isn't a correlation between measured performance and sound quality/signal fidelity.
There really is, but that just isn't your preference.

Do things your way... It is fine to have a preference.
'Sounds good' in your case is actually 'sounds good to me' for you but that is because of your preference not because a complete set of measurements (which is almost never done, also not here) does not correlate to signal fidelity.

 

[jtgofish]

Well it is actually more like preferences plural.Applied to four different system in two different rooms.I gave up chasing "perfection" years ago but I have certainly been down the accuracy road .I just found it awfully boring and restricting after a while.I have friends who have gone a long way down that accuracy road only to end up hardly ever listening to their systems. I guess they get some satisfaction from being able to tell themselves they own one the worlds best measuring,lowest distortion speakers though.Even if they hardly ever listen to them.

 

[SIY]

Do any of those "authorities" have experience in setting up controlled sensory tests? The vast majority of engineers do not.

 

[Dimitrov]

Have you guys ever heard stories where a non-audiophile family members immediately picked up on a difference in sound, without having been told that cables had been changed (or even been aware that the system was being worked on)?

I've heard those stories several times, as if that were some kind of proof.

 

[BDWoody]

I am sure the listeners in the group test that I still have would appreciate your thoughts on their professionalism and suitability for the job.

Do any of them have any background in sensory testing? Did they seek help from someone who does/did?

You are minimizing the importance of meaningful controls, like most everyone who sells based on a story.

Edit: @SIY got there first.

 

[Mart68]

Well it is actually more like preferences plural.Applied to four different system in two different rooms.I gave up chasing "perfection" years ago but I have certainly been down the accuracy road .I just found it awfully boring and restricting after a while.I have friends who have gone a long way down that accuracy road only to end up hardly ever listening to their systems. I guess they get some satisfaction from being able to tell themselves they own one the worlds best measuring,lowest distortion speakers though.Even if they hardly ever listen to them.

You buy the world's most accurate speakers then EQ them to your taste.

You've gone off-piste anyway, no-one here is arguing against personal preference. There's easer ways to get the sound you prefer than buying speakers and other equipment that measure badly. Usually, cheaper too since these days the worst kit tends to be very expensive.

 

[Spkrdctr]

I don't really believe in those tests, sorry. https://www.stereophile.com/content/blind-testing-flaws-1 and i respectfully disagree with this as well "But "vinyl", "visceral and real", and "spatial presentation" is not something that can be used to distinguish audio gear". No offense but it makes me feel like you are not even listening to your gear.

I read some of the posts on your link. One, it sounds good to someone who has no knowledge at all about sound. Second, what they are posting has been proven wrong thousands of times in the last 50 years. I mean it it. Everything these guys are posting is wrong and proven wrong. So many incorrect assumptions are made. If is like trying to tell someone they can't shoot a gun and hit a full moon that is just sitting right there in the sky. Hitting it sounds good if you know nothing of the science involved. But, trying to educate them is a gigantic job. To learn about the facts and science of audio, if they just hang out here on ASR they will learn over time. Remember ALL of the electronic equipment was designed by engineers using electrical science that is proven (no snake oil products).
So in the end you have 30 people posting idiotic posts on sound equipment and listening tests. None of them have any idea of the science involved. But if you try to teach them the truth they go into anti-science mode and tell you your crazy and have tin ears etc.

 

[solderdude]

.I have friends who have gone a long way down that accuracy road only to end up hardly ever listening to their systems.

They told you this or you assume they do ?

I guess they get some satisfaction from being able to tell themselves they own one the worlds best measuring,lowest distortion speakers though.

Your guess though.

Even if they hardly ever listen to them.

They told you they hardly ever do or do you assume they hardly ever do.

 

[Spkrdctr]

I guess I'm done. I do not have it in me to have an argument with anti-science people anymore. If they want to remain dumb, who am I to tell them different. They have no interest in learning, just arguing idiotic stupid ideas endlessly. I leave that to Amir, he is the next generation educator of these people. Go Amir go.

 

[Doodski]

The more you try to help them, the more resistant they become.

It's as if they're scared of ...... something.

Scared of being duped and misled. There are so many opinions regarding audio gear that it must be a minefield of ideas to try and walk through.

 

[Doodski]

You are so right! That's why a learning experience needs to be based on science and logic. Get the basics down. Learn what discovery and experimentation really amount to. Get a foundation in reality. Then you won't be scared of being duped and misled, because you will have the tools that you need to navigate the minefield of "opinions".

After that, where you go and what you do is entirely up to you.

Jim

About age 26 I had enough of the minefield as I was a AV Stereo Salesperson for 9 years and was veryy burned out on people. So I studied electronics at a technical institute and got off that merry go round of ups and downs of sales. I traded one merry go round for one very serious study that was a bit painful at times due to the course load but now that it's all over and I've experience beaucoup it doesn't look so bad. I try and impart as much of that knowledge as is possible upon the peeps that come here looking for guidance and truth. Sometimes it's a bit repetitive but it's not as bad as it seems. It's actually most days pretty easy to do.