Master Thread: Are measurements Everything or Nothing?

[Purité Audio]

When you say "entirely characterize the component" do you think that it's possible to predict the preferred component for a given listener using its measurements?

For example: If a given listener were to audition 10 different DACs in their system, do you think it would be possible to accurately predict their preference between 2 additional DACs based solely on the measurements?

A components measurements just show how well the unit has been engineered or not, some may prefer audible distortion.
One has to be careful as many report differences when of course they don’t exist and the measurements would verify this.
Keith

 

[blackdiamond]

A DAC does not output sound. It only outputs an electrical signal, which (usually) goes into an amplifier. Everything that matters is in that electrical signal. And these days we have the technology to measure that electrical signal with super high precision -- much higher than what our ears allow us to do.

[Edit] We can easily have DAC's these days that are basically perfect. If two DAC's are both perfect, you will not be able to tell them apart. Since you cannot tell them apart, there will be no preference. They are equal. If 2 DAC's are audibly different, at least one of them is broken.

Have you had opportunity to test your theory that two DACs that measure "identically" sound the same in practice? I don't disagree with the theoretical aspect of this, but I'd love the opportunity to have two DACs that measure "identically" tested empirically to see if people can tell them apart with any reliability?

 

[NTK]

Have you had opportunity to test your theory that two DACs that measure "identically" sound the same in practice? I don't disagree with the theoretical aspect of this, but I'd love the opportunity to have two DACs that measure "identically" tested empirically to see if people can tell them apart with any reliability?

I am a science believer, and I know first hand what modern scientific instruments can do. Plenty of other people have done it. I cannot think of a reason why I need to.

I believe Antarctica exit. And no, I have never been there. And for that matter, Africa too.

 

[blackdiamond]

A components measurements just show how well the unit has been engineered or not, some may prefer audible distortion.
One has to be careful as many report differences when of course they don’t exist and the measurements would verify this.
Keith

I do think that placebo and expectation bias can be very difficult to eliminate when making comparisons with audio equipment. In my experience, I find that it's very difficult to do A/B testing for a variety of reasons that range from objective things like level matching to more subjective things like over analyzing things. One example from my experience was Nordost Sort Kones that I put quite a bit of effort into trying to determine if I could hear a difference with them under my SACD player. In the end, I was never convinced that I could tell if they were there or not but also recognize that my system was not up to the challenge so have not completely discounted that they could make a difference in a different system. In the cases where I do consider that there is a difference they have been significant to my ears and not something that I needed to do comparisons to confirm.

What changes have you made in your system that have heard a difference with?

 

[blackdiamond]

I am a science believer, and I know first hand what modern scientific instruments can do. Plenty of other people have done it. I cannot think of a reason why I need to.

I believe Antarctica exit. And no, I have never been there. And for that matter, Africa too.

Where's the fun in being an audiophile without curiosity? I also believe in science. My starting point is much closer to it's just 1s and 0s than the other extreme, but my own experience has allowed me to hear things in my system that I wasn't expecting. The result is that I've become more and more curious to test things for myself.

I started with a $40 power cord because I was curious. I had mixed results. For one component the difference was significant. For the others I couldn't tell any difference. I tried a more expensive (used) power cord for my DAC and still didn't hear any obvious difference.

I started with a $100 analog interconnect because I was curious. I found it to make a nice improvement, so I moved up to a $200 analog interconnect of the same brand (to minimize the variables) and again found it to be a step up though not as significant as the first.

I started with a $100 digital interconnect because I was curious. I found that I couldn't tell any difference in my system.

I changed the op amps in my old DAC because I was curious. I found that it made a significant difference. The DAC has parallel solid state and tube buffer outputs. I preferred the solid state output with the original op amps and then preferred the tube buffered output after. I rolled a tube in the same DAC and didn't hear any obvious difference.

I'm considering a power supply upgrade for my Bluesound Node 2i because I'm curious. I've not yet pulled the trigger because I'm not positive that I should even expect to hear a difference/improvement.

Science is all about experimentation. Audiophiles are all over the map with their opinions, so I'm always looking for the commonalities that may lead to a grain of truth. I've seen a lot of audiophiles attribute a particular change to a single variable when multiple variables were changed, so I know that most things cannot be taken at face value.

I know what I've heard in my system with my ears and also understand that there are likely audiophiles with better ears and certainly better systems.

From a place of genuine curiosity to better understand your experience/perspective, are you willing to share your setup and what changes you have made that felt made a difference? Has anything ever surprised you in your system?

 

[Blumlein 88]

Here is a surprise I've heard. Was trying out a new DAC with friends. Hooked both DACs up, listened to some short tracks for reference, switched to the new DAC listened to the same tracks. We heard how the new DAC was different, even agreed mostly what those differences were and which we preferred. Decided to go back and hear the old one just to be sure. Found we had mistakenly listened to the old DAC the whole time and never switched. I have other examples.

First off, if you don't match levels your listening comparisons/impressions are useless and meaningless.

You might be surprised all the ways we can convince ourselves we hear differences when none are there. That is not to say no differences exist. Only that one has to do controlled listening to have a chance. One also can learn when some claim is simply way beyond credibility and not worth entertaining.

 

[blackdiamond]

Science is all about CONTROLLED experimentation.

Jim

Fair. And I do recognize that it's very difficult to control all of the variables in the audiophile world.

What I find most reliable for me is listening to my system and becoming very familiar with how it sounds rather than doing A/B comparisions. When I do make a change, I can almost always tell if there's a difference immediately but it's confirmed when the music that I listen to becomes different. I've found with most of my notable upgrades that my favorite albums or songs on an album change.

Is not being able to completely control the experimentation a reason to not do it at all?

 

[DonR]

Is not being able to completely control the experimentation a reason to not do it at all?

yes

 

[NTK]

Has anything ever surprised you in your system?

Yes. Many years ago I had my collection of CD ripped to 256 kbps MP3 (not even with Lame). When storage became more plentiful, I redid them to FLAC. You know how the story goes. I was marveling at the improved sound quality of my new FLAC collection, while not noticing that I was still playing my files from the old library.

 

[blackdiamond]

Here is a surprise I've heard. Was trying out a new DAC with friends. Hooked both DACs up, listened to some short tracks for reference, switched to the new DAC listened to the same tracks. We heard how the new DAC was different, even agreed mostly what those differences were and which we preferred. Decided to go back and hear the old one just to be sure. Found we had mistakenly listened to the old DAC the whole time and never switched. I have other examples.

First off, if you don't match levels your listening comparisons/impressions are useless and meaningless.

You might be surprised all the ways we can convince ourselves we hear differences when none are there. That is not to say no differences exist. Only that one has to do controlled listening to have a chance. One also can learn when some claim is simply way beyond credibility and not worth entertaining.

This is a great example of why I don't spend much time doing A/B comparisons. I find that it leads to over thinking things which can lead to exactly what you describe.

I recently took my Morrow Audio MA4 balanced interconnects over to a friends house to try out in his system. We did not have much time so we listened to a few songs with his cables and then switched to my cables. We both noted an improvement and described what we heard in similar ways. What was surprising was when he put on a song with my cables that we hadn't listened to yet. I immediately was really impressed with the way it sounded and his reaction was simply to say that, "it had never sounded better." To finalize the brief comparison, we switched back and ironically, the song that had sounded so great was the definition of underwhelming. I found it ironic that the songs we had specifically chosen for comparison turned out to not show the differences nearly as much as the somewhat random one did.

 

[blackdiamond]

Yes. Many years ago I had my collection of CD ripped to 256 kbps MP3 (not even with Lame). When storage became more plentiful, I redid them to FLAC. You know how the story goes. I was marveling at the improved sound quality of my new FLAC collection, while not noticing that I was still playing my files from the old library.

It sounds like you've had some experiences where you became "victim" to placebo/confirmation bias and concluded that it is the explanation for the vast majority of differences that audiophiles report that they can hear.

What changes in an audio system do your expect to result in an audible difference? Where do you think that it all becomes placebo and expectation bias?

 

[Blumlein 88]

Fair. And I do recognize that it's very difficult to control all of the variables in the audiophile world.

What I find most reliable for me is listening to my system and becoming very familiar with how it sounds rather than doing A/B comparisions. When I do make a change, I can almost always tell if there's a difference immediately but it's confirmed when the music that I listen to becomes different. I've found with most of my notable upgrades that my favorite albums or songs on an album change.

Is not being able to completely control the experimentation a reason to not do it at all?

Your methodology is the one MOST likely to mislead you. It feels the reverse. I've experienced exactly what you describe. You vary levels, your moods vary, noise levels vary, and you'll rarely discern anything real this way. I get it, I was there once years ago.

The most discerning method is A vs B with levels matched and switching done very quickly. Switching that takes more than a few seconds rapidly erodes the ability to hear differences. Quite often you'll hear none. Go listen over a few days and get familiar and you'll feel confident of differences you hear. The first method is usually correct, and the latter long term method rarely is. You'll build a whole set of experiences reinforcing what you feel about it like changing music you listen too. It is all a mirage.

 

[blackdiamond]

Your methodology is the one MOST likely to mislead you. It feels the reverse. I've experienced exactly what you describe. You vary levels, your moods vary, noise levels vary, and you'll rarely discern anything real this way. I get it, I was there once years ago.

The most discerning method is A vs B with levels matched and switching done very quickly. Switching that takes more than a few seconds rapidly erodes the ability to hear differences. Quite often you'll hear none. Go listen over a few days and get familiar and you'll feel confident of differences you hear. The first method is usually correct, and the latter long term method rarely is. You'll build a whole set of experiences reinforcing what you feel about it like changing music you listen too. It is all a mirage.

This is an interesting concept. I won't disagree with you that A vs B might be required to discern subtle differences. My philosophy is that if I have to spend hours carefully comparing A vs B then the result isn't worth the effort. On the flip side, I'd describe many of the changes that I've heard as being clear and obvious. In these cases, I think it's expected that having a strong point of reference from being familiar with my system would make it easy to hear if something changed significantly.

One time that I nearly tricked myself was when I purchased a Schiit Modi Multibit DAC. I immediately noticed a difference and instinctively considered it to be an improvement. After listening to it for a couple of weeks and comparing it to my previous DAC I was close to keeping it. Then I remembered that I was only comparing it to the solid state output on my previous DAC. When I compared it to the tube buffered output on my previous DAC I honestly couldn't tell any difference. Ironically, I had not even kept the tube buffered output connected because I had such a strong preference for the solid state output. Once I realized that I had been duped, I sent the Schiit DAC back. This is an example that demonstrates that I did hear a difference, but my perception of that difference was where I was wrong.

 

[sonitus mirus]

This is an interesting concept. I won't disagree with you that A vs B might be required to discern subtle differences. My philosophy is that if I have to spend hours carefully comparing A vs B then the result isn't worth the effort. On the flip side, I'd describe many of the changes that I've heard as being clear and obvious. In these cases, I think it's expected that having a strong point of reference from being familiar with my system would make it easy to hear if something changed significantly.

One time that I nearly tricked myself was when I purchased a Schiit Modi Multibit DAC. I immediately noticed a difference and instinctively considered it to be an improvement. After listening to it for a couple of weeks and comparing it to my previous DAC I was close to keeping it. Then I remembered that I was only comparing it to the solid state output on my previous DAC. When I compared it to the tube buffered output on my previous DAC I honestly couldn't tell any difference. Ironically, I had not even kept the tube buffered output connected because I had such a strong preference for the solid state output. Once I realized that I had been duped, I sent the Schiit DAC back. This is an example that demonstrates that I did hear a difference, but my perception of that difference was where I was wrong.

You are not hearing any difference between DACs unless it is trivially obvious with measurements. That fact is not debatable without a significant amount of proof. No amount of anecdotal evidence will ever suffice.

 

[funnychap]

Seems like so many review threads get challenged with:

1. Measurements are not everything.

2. You all never listen.

3. I trust my ears, not graphs.

4. I don't listen to graphs. I listen to music.

5. You all must not listen to music at all.

6. Why don't you all buy the best SINAD gear?

7. I have heard your best SINAD gear and they sound terrible. I don't like any of this Chinese stuff.

8. You don't trust your ears. I/we do.

9. All these reviewers/youtubers/audophiles say these amps, DACs, etc. sound different and you say they don't. They can't all be wrong.

10. Surely designers have created certain house sound for each equipment which your measurements don't show.

11. Your measurements are only at one frequency. You need to also measure X, Y and Z like impulse response, slew rate, etc., etc.

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.

On and on...

I have had to answer these so many times that I thought it is time to stop having them go into every review as they are not product specific. From here on, any such questions should be posted here. Answers will be given in this thread and simply referenced in future challenges in other threads.

@AdamG247 and @BDWoody, please direct any future posts in review threads to here and not allow discussions there.

Thanks. You all are free to discuss this topic, provide answers, argue, whatever, in this thread.

3. I trust my ears, not graphs.

 

[Ken1951]

Jeez, Louise.....

 

[Xulonn]

Wash, rinse, recycle, repeat...

 

[blackdiamond]

You are not hearing any difference between DACs unless it is trivially obvious with measurements. That fact is not debatable without a significant amount of proof. No amount of anecdotal evidence will ever suffice.

So, I post that I heard the difference between a solid state output DAC A and a tube buffered output on DAC A and that I found that DAC B sounded identical to the tube buffered output of DAC A and this is your response? Would it not be expected that the tube buffer would change the way the output measured?

I've seen people say that our ability to measure exceed our ability to hear. What I've not seen is someone that can correlate measurements to how a component will sound.

Are you familiar with Iconoclast cables that are designed by Galen Gareis? I love his scientific approach to design and find it very interesting that he offers multiple conductor materials even though he makes it very clear that he does not know how to design for them. The point is that he can hear something that he doesn't know how to measure or design for. He seems like someone that should know as much from a scientific design perspective as anyone in the industry. If he doesn't think that measurements tell the whole story then it seems to be at least plausable.

How does someone "know" if a particular set of measurements for a given thing are sufficient to account for everything without testing things and seeing if the measurements can explain everything that has been experienced?

 

[blackdiamond]

1.) Yes. You may well mislead yourself, believing your results to be worthwhile, factual or correct when they are not. You may even misidentify the nature of your experiment, causing you to embark on a totally fruitless quest.

2.) You may then mislead others. I don't mean mislead out of malice, I mean mislead others due to a confidence that is unfounded.

Witness the "cold fusion" debacle of a few years back. Only when controls were examined in greatest detail was there a realization that the "breakthrough" did not exist. I know that the enjoyment of audio doesn't rank up there with the importance of cold fusion, but the principles of disciplined examination and cross-examination are the same.

Jim

How does someone use measurements if they never do anything to develop a knowledge of what they mean in a practical sense? I agree that if the conclusions drawn are overly specific they will almost certainly be misleading. For example, I've seen people make statements about how a larger conductor gauge in speaker cables made a specific difference in their system when the cables being compared had nothing in common beyond being long and having two ends.

I had a Krell S-300i amplifier that I was comparing to my Pathos Classic One Mk III and they very sounded different to me, but I really struggled to choose which one was better. Then I rolled the tubes in the Pathos based on a recommendation and the transformation was significant enough that I sold the Krell soon after and have never looked back. How would I ever be able to present a measurement for this? What I learned was that the potential improvement from tube rolling was much more significant that I expected to be possible. Assuming that I had access to get every available measurement of the Pathos before and after, would there be any way use that information to identify another amplifier that I might like more?

 

[blackdiamond]

One thing that I'll say is that I'd like nothing more (maybe a slight stretch) than to sit down and listen with several of you to better understand your perspective and experiences. I'm fascinated at the broad spectrum of audiophiles and I enjoy trying to understand the different approaches. When it comes to the "measurements rule" perspective, I have yet to understand how the approach translates into evaluating audio equipment. It seems that it should result in way less turnover in equipment compared to the "trust your ears" approach where literally everything matters.