Master Thread: Are measurements Everything or Nothing?

[blackdiamond]

I'm a member on Audiogon where the most common view is that "everything matters" to a point where nothing matters because it becomes impossible to isolate any particular factor. For example, I've read that disconnecting and reconnecting cables can improve the sound (i.e. maybe restoring a solid contact) while I've also read that cables need time to setting after being disturbed. It seems that both cannot be true.

Throughout my audiophile journey I've tried to keep an open mind while looking for opportunities to test things out for myself. I also try to be open to the fact that while my system is fantastic by any normal standard, it's no where near the pinnacle of audio technology so there may be things that I cannot experience with my system. Over time, I have become even more open minded based on what I have heard in my system. Here's a brief synopsis of my experience:

Tubes
I have a Pathos Classic One Mk III integrated amplifier and it was simply transformed when I rolled in some 1960s Mullard Tubes. Tubes changing the sound of components isn't really a debated topic, but I was really impressed with how significant the change was.

Cables
My experience with cables started with a couple of Waudio power cables. The power cable on my integrated amplifier made a significant difference to my ear. I moved the second power cable around among my SACD player (bad), power conditioner (neutral), and DAC (neutral). I currently have Pangea power cord on my DAC with neutral results.

After power cables, I tried interconnects. I had previously used Blue Jeans Cables, so started with Audioquest Chicago (RCA) and really liked the change in sound. I opted to move up again to the Audioquest Sydney (RCA) and felt that it maintained the same sound characteristic while improving the overall sound. I stopped there until I got a new DAC (a huge improvement in my system) where I was able to move to balanced connections. I opted to go with Morrow Audio MA4 (XLR) cables and once again found them to be a significant improvement. I did try an Audioquest Cinnamon digital coax, but heard no difference in comparison to my Blue Jeans Cable.

I also tried some well reviewed (and cheap) speaker cables and found them to be different, but not in a positive way. I even loaned them to a friend that had a very similar experience with them. We were both "relieved" to go back to our previous speaker cables. It may be worth noting that we both have twisted speakers cables (I have Blue Jeans Cable 10 AWG) and the ones we tried used parallel conductors and we both felt that there was a loss of detail.

Tweaks
I've not heard any differences from things like isolation devices, though I did play with some Nordost Sort Kones over the years but never could convince myself that there was anything beyond placebo and/or expectation bias in my system.

I'm currently looking at a power supply upgrade for my Bluesound Node 2i for a different experience.

Recognizing that this is a bit of a ramble, what differences have you heard in your system and what things have your tried where you didn't hear any difference?

 

[blackdiamond]

Measurements entirely characterise the component, that is not to say an individual may not prefer a poorer measuring product, if indeed those poorer measurements lead to an audible difference.
‘Keith

When you say "entirely characterize the component" do you think that it's possible to predict the preferred component for a given listener using its measurements?

For example: If a given listener were to audition 10 different DACs in their system, do you think it would be possible to accurately predict their preference between 2 additional DACs based solely on the measurements?

 

[NTK]

When you say "entirely characterize the component" do you think that it's possible to predict the preferred component for a given listener using its measurements?

For example: If a given listener were to audition 10 different DACs in their system, do you think it would be possible to accurately predict their preference between 2 additional DACs based solely on the measurements?

A DAC does not output sound. It only outputs an electrical signal, which (usually) goes into an amplifier. Everything that matters is in that electrical signal. And these days we have the technology to measure that electrical signal with super high precision -- much higher than what our ears allow us to do.

[Edit] We can easily have DAC's these days that are basically perfect. If two DAC's are both perfect, you will not be able to tell them apart. Since you cannot tell them apart, there will be no preference. They are equal. If 2 DAC's are audibly different, at least one of them is broken.

 

[Purité Audio]

When you say "entirely characterize the component" do you think that it's possible to predict the preferred component for a given listener using its measurements?

For example: If a given listener were to audition 10 different DACs in their system, do you think it would be possible to accurately predict their preference between 2 additional DACs based solely on the measurements?

A components measurements just show how well the unit has been engineered or not, some may prefer audible distortion.
One has to be careful as many report differences when of course they don’t exist and the measurements would verify this.
Keith

 

[blackdiamond]

A DAC does not output sound. It only outputs an electrical signal, which (usually) goes into an amplifier. Everything that matters is in that electrical signal. And these days we have the technology to measure that electrical signal with super high precision -- much higher than what our ears allow us to do.

[Edit] We can easily have DAC's these days that are basically perfect. If two DAC's are both perfect, you will not be able to tell them apart. Since you cannot tell them apart, there will be no preference. They are equal. If 2 DAC's are audibly different, at least one of them is broken.

Have you had opportunity to test your theory that two DACs that measure "identically" sound the same in practice? I don't disagree with the theoretical aspect of this, but I'd love the opportunity to have two DACs that measure "identically" tested empirically to see if people can tell them apart with any reliability?

 

[NTK]

Have you had opportunity to test your theory that two DACs that measure "identically" sound the same in practice? I don't disagree with the theoretical aspect of this, but I'd love the opportunity to have two DACs that measure "identically" tested empirically to see if people can tell them apart with any reliability?

I am a science believer, and I know first hand what modern scientific instruments can do. Plenty of other people have done it. I cannot think of a reason why I need to.

I believe Antarctica exit. And no, I have never been there. And for that matter, Africa too.

 

[blackdiamond]

A components measurements just show how well the unit has been engineered or not, some may prefer audible distortion.
One has to be careful as many report differences when of course they don’t exist and the measurements would verify this.
Keith

I do think that placebo and expectation bias can be very difficult to eliminate when making comparisons with audio equipment. In my experience, I find that it's very difficult to do A/B testing for a variety of reasons that range from objective things like level matching to more subjective things like over analyzing things. One example from my experience was Nordost Sort Kones that I put quite a bit of effort into trying to determine if I could hear a difference with them under my SACD player. In the end, I was never convinced that I could tell if they were there or not but also recognize that my system was not up to the challenge so have not completely discounted that they could make a difference in a different system. In the cases where I do consider that there is a difference they have been significant to my ears and not something that I needed to do comparisons to confirm.

What changes have you made in your system that have heard a difference with?

 

[blackdiamond]

I am a science believer, and I know first hand what modern scientific instruments can do. Plenty of other people have done it. I cannot think of a reason why I need to.

I believe Antarctica exit. And no, I have never been there. And for that matter, Africa too.

Where's the fun in being an audiophile without curiosity? I also believe in science. My starting point is much closer to it's just 1s and 0s than the other extreme, but my own experience has allowed me to hear things in my system that I wasn't expecting. The result is that I've become more and more curious to test things for myself.

I started with a $40 power cord because I was curious. I had mixed results. For one component the difference was significant. For the others I couldn't tell any difference. I tried a more expensive (used) power cord for my DAC and still didn't hear any obvious difference.

I started with a $100 analog interconnect because I was curious. I found it to make a nice improvement, so I moved up to a $200 analog interconnect of the same brand (to minimize the variables) and again found it to be a step up though not as significant as the first.

I started with a $100 digital interconnect because I was curious. I found that I couldn't tell any difference in my system.

I changed the op amps in my old DAC because I was curious. I found that it made a significant difference. The DAC has parallel solid state and tube buffer outputs. I preferred the solid state output with the original op amps and then preferred the tube buffered output after. I rolled a tube in the same DAC and didn't hear any obvious difference.

I'm considering a power supply upgrade for my Bluesound Node 2i because I'm curious. I've not yet pulled the trigger because I'm not positive that I should even expect to hear a difference/improvement.

Science is all about experimentation. Audiophiles are all over the map with their opinions, so I'm always looking for the commonalities that may lead to a grain of truth. I've seen a lot of audiophiles attribute a particular change to a single variable when multiple variables were changed, so I know that most things cannot be taken at face value.

I know what I've heard in my system with my ears and also understand that there are likely audiophiles with better ears and certainly better systems.

From a place of genuine curiosity to better understand your experience/perspective, are you willing to share your setup and what changes you have made that felt made a difference? Has anything ever surprised you in your system?

 

[Blumlein 88]

Here is a surprise I've heard. Was trying out a new DAC with friends. Hooked both DACs up, listened to some short tracks for reference, switched to the new DAC listened to the same tracks. We heard how the new DAC was different, even agreed mostly what those differences were and which we preferred. Decided to go back and hear the old one just to be sure. Found we had mistakenly listened to the old DAC the whole time and never switched. I have other examples.

First off, if you don't match levels your listening comparisons/impressions are useless and meaningless.

You might be surprised all the ways we can convince ourselves we hear differences when none are there. That is not to say no differences exist. Only that one has to do controlled listening to have a chance. One also can learn when some claim is simply way beyond credibility and not worth entertaining.

 

[blackdiamond]

Science is all about CONTROLLED experimentation.

Jim

Fair. And I do recognize that it's very difficult to control all of the variables in the audiophile world.

What I find most reliable for me is listening to my system and becoming very familiar with how it sounds rather than doing A/B comparisions. When I do make a change, I can almost always tell if there's a difference immediately but it's confirmed when the music that I listen to becomes different. I've found with most of my notable upgrades that my favorite albums or songs on an album change.

Is not being able to completely control the experimentation a reason to not do it at all?

 

[DonR]

Is not being able to completely control the experimentation a reason to not do it at all?

yes

 

[NTK]

Has anything ever surprised you in your system?

Yes. Many years ago I had my collection of CD ripped to 256 kbps MP3 (not even with Lame). When storage became more plentiful, I redid them to FLAC. You know how the story goes. I was marveling at the improved sound quality of my new FLAC collection, while not noticing that I was still playing my files from the old library.

 

[blackdiamond]

Here is a surprise I've heard. Was trying out a new DAC with friends. Hooked both DACs up, listened to some short tracks for reference, switched to the new DAC listened to the same tracks. We heard how the new DAC was different, even agreed mostly what those differences were and which we preferred. Decided to go back and hear the old one just to be sure. Found we had mistakenly listened to the old DAC the whole time and never switched. I have other examples.

First off, if you don't match levels your listening comparisons/impressions are useless and meaningless.

You might be surprised all the ways we can convince ourselves we hear differences when none are there. That is not to say no differences exist. Only that one has to do controlled listening to have a chance. One also can learn when some claim is simply way beyond credibility and not worth entertaining.

This is a great example of why I don't spend much time doing A/B comparisons. I find that it leads to over thinking things which can lead to exactly what you describe.

I recently took my Morrow Audio MA4 balanced interconnects over to a friends house to try out in his system. We did not have much time so we listened to a few songs with his cables and then switched to my cables. We both noted an improvement and described what we heard in similar ways. What was surprising was when he put on a song with my cables that we hadn't listened to yet. I immediately was really impressed with the way it sounded and his reaction was simply to say that, "it had never sounded better." To finalize the brief comparison, we switched back and ironically, the song that had sounded so great was the definition of underwhelming. I found it ironic that the songs we had specifically chosen for comparison turned out to not show the differences nearly as much as the somewhat random one did.

 

[blackdiamond]

Yes. Many years ago I had my collection of CD ripped to 256 kbps MP3 (not even with Lame). When storage became more plentiful, I redid them to FLAC. You know how the story goes. I was marveling at the improved sound quality of my new FLAC collection, while not noticing that I was still playing my files from the old library.

It sounds like you've had some experiences where you became "victim" to placebo/confirmation bias and concluded that it is the explanation for the vast majority of differences that audiophiles report that they can hear.

What changes in an audio system do your expect to result in an audible difference? Where do you think that it all becomes placebo and expectation bias?

 

[Blumlein 88]

Fair. And I do recognize that it's very difficult to control all of the variables in the audiophile world.

What I find most reliable for me is listening to my system and becoming very familiar with how it sounds rather than doing A/B comparisions. When I do make a change, I can almost always tell if there's a difference immediately but it's confirmed when the music that I listen to becomes different. I've found with most of my notable upgrades that my favorite albums or songs on an album change.

Is not being able to completely control the experimentation a reason to not do it at all?

Your methodology is the one MOST likely to mislead you. It feels the reverse. I've experienced exactly what you describe. You vary levels, your moods vary, noise levels vary, and you'll rarely discern anything real this way. I get it, I was there once years ago.

The most discerning method is A vs B with levels matched and switching done very quickly. Switching that takes more than a few seconds rapidly erodes the ability to hear differences. Quite often you'll hear none. Go listen over a few days and get familiar and you'll feel confident of differences you hear. The first method is usually correct, and the latter long term method rarely is. You'll build a whole set of experiences reinforcing what you feel about it like changing music you listen too. It is all a mirage.

 

[blackdiamond]

Your methodology is the one MOST likely to mislead you. It feels the reverse. I've experienced exactly what you describe. You vary levels, your moods vary, noise levels vary, and you'll rarely discern anything real this way. I get it, I was there once years ago.

The most discerning method is A vs B with levels matched and switching done very quickly. Switching that takes more than a few seconds rapidly erodes the ability to hear differences. Quite often you'll hear none. Go listen over a few days and get familiar and you'll feel confident of differences you hear. The first method is usually correct, and the latter long term method rarely is. You'll build a whole set of experiences reinforcing what you feel about it like changing music you listen too. It is all a mirage.

This is an interesting concept. I won't disagree with you that A vs B might be required to discern subtle differences. My philosophy is that if I have to spend hours carefully comparing A vs B then the result isn't worth the effort. On the flip side, I'd describe many of the changes that I've heard as being clear and obvious. In these cases, I think it's expected that having a strong point of reference from being familiar with my system would make it easy to hear if something changed significantly.

One time that I nearly tricked myself was when I purchased a Schiit Modi Multibit DAC. I immediately noticed a difference and instinctively considered it to be an improvement. After listening to it for a couple of weeks and comparing it to my previous DAC I was close to keeping it. Then I remembered that I was only comparing it to the solid state output on my previous DAC. When I compared it to the tube buffered output on my previous DAC I honestly couldn't tell any difference. Ironically, I had not even kept the tube buffered output connected because I had such a strong preference for the solid state output. Once I realized that I had been duped, I sent the Schiit DAC back. This is an example that demonstrates that I did hear a difference, but my perception of that difference was where I was wrong.

 

[sonitus mirus]

This is an interesting concept. I won't disagree with you that A vs B might be required to discern subtle differences. My philosophy is that if I have to spend hours carefully comparing A vs B then the result isn't worth the effort. On the flip side, I'd describe many of the changes that I've heard as being clear and obvious. In these cases, I think it's expected that having a strong point of reference from being familiar with my system would make it easy to hear if something changed significantly.

One time that I nearly tricked myself was when I purchased a Schiit Modi Multibit DAC. I immediately noticed a difference and instinctively considered it to be an improvement. After listening to it for a couple of weeks and comparing it to my previous DAC I was close to keeping it. Then I remembered that I was only comparing it to the solid state output on my previous DAC. When I compared it to the tube buffered output on my previous DAC I honestly couldn't tell any difference. Ironically, I had not even kept the tube buffered output connected because I had such a strong preference for the solid state output. Once I realized that I had been duped, I sent the Schiit DAC back. This is an example that demonstrates that I did hear a difference, but my perception of that difference was where I was wrong.

You are not hearing any difference between DACs unless it is trivially obvious with measurements. That fact is not debatable without a significant amount of proof. No amount of anecdotal evidence will ever suffice.

 

[funnychap]

Seems like so many review threads get challenged with:

1. Measurements are not everything.

2. You all never listen.

3. I trust my ears, not graphs.

4. I don't listen to graphs. I listen to music.

5. You all must not listen to music at all.

6. Why don't you all buy the best SINAD gear?

7. I have heard your best SINAD gear and they sound terrible. I don't like any of this Chinese stuff.

8. You don't trust your ears. I/we do.

9. All these reviewers/youtubers/audophiles say these amps, DACs, etc. sound different and you say they don't. They can't all be wrong.

10. Surely designers have created certain house sound for each equipment which your measurements don't show.

11. Your measurements are only at one frequency. You need to also measure X, Y and Z like impulse response, slew rate, etc., etc.

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.

On and on...

I have had to answer these so many times that I thought it is time to stop having them go into every review as they are not product specific. From here on, any such questions should be posted here. Answers will be given in this thread and simply referenced in future challenges in other threads.

@AdamG247 and @BDWoody, please direct any future posts in review threads to here and not allow discussions there.

Thanks. You all are free to discuss this topic, provide answers, argue, whatever, in this thread.

3. I trust my ears, not graphs.

 

[Ken1951]

Jeez, Louise.....

 

[Xulonn]

Wash, rinse, recycle, repeat...