Master Thread: Are Measurements Everything or Nothing?

[Jazigo]

I will try to enter this debate as gentle as I can, with a point in favor of the subjectivists. Please bear with me as I walk on a shady ground.

As far as every measurement done, it is flawless and outperforms every human organ. Make no mistake about that.

To this day, I have never been able to discern between dacs (the delta-sigma ones) and I suspect that I probably never will.
On the other hand, every speaker I try is unique, even if they measure quite similar. There are of course degrees to these differences.

So far I think most of us can agree, but where I learned my lesson was with amplifiers. The simplest block in the signal chain. The PA5 II vs B100. I simply though that there would be no differences what so ever, but it was apparent immediately that there was a difference. I got schooled and did my own measurements, but they reveled no differences. So I chalked it up to my own bias and moved on with my life.

B100 died, so I had to go back to my PA5 II, and holy hell, no one can tell me that there is no difference. As I stand, I think you have to deaf to not actually hear the difference.

This puts me in an a conundrum: I have seen the measurements, done my own measurements, convinced myself that my brain is a bias machine, but here I stand.
Personally, I think that for amps there probably is a measurement that isn't done that would reveal this. I tried to do some research and perhaps there is something to the lack of bandwidth in a class-d amplifier that does it?

This video here kind of catches what I hear between a PA5 II and B100s:

With all that said and done. In favor of the objectivists here: Everything I have heard that measure goods here on the forum is neutral and very well behaved. So at the moment, I use the measurements to guarantee that the "ingredients" and the "kitchen" are in order and that the "flavors" are balanced, but there are still areas not covered.

 

[Blockader]

I will try to enter this debate as gentle as I can, with a point in favor of the subjectivists. Please bear with me as I walk on a shady ground.

As far as every measurement done, it is flawless and outperforms every human organ. Make no mistake about that.

To this day, I have never been able to discern between dacs (the delta-sigma ones) and I suspect that I probably never will.
On the other hand, every speaker I try is unique, even if they measure quite similar. There are of course degrees to these differences.

So far I think most of us can agree, but where I learned my lesson was with amplifiers. The simplest block in the signal chain. The PA5 II vs B100. I simply though that there would be no differences what so ever, but it was apparent immediately that there was a difference. I got schooled and did my own measurements, but they reveled no differences. So I chalked it up to my own bias and moved on with my life.

B100 died, so I had to go back to my PA5 II, and holy hell, no one can tell me that there is no difference. As I stand, I think you have to deaf to not actually hear the difference.

This puts me in an a conundrum: I have seen the measurements, done my own measurements, convinced myself that my brain is a bias machine, but here I stand.
Personally, I think that for amps there probably is a measurement that isn't done that would reveal this. I tried to do some research and perhaps there is something to the lack of bandwidth in a class-d amplifier that does it?

This video here kind of catches what I hear between a PA5 II and B100s:

With all that said and done. In favor of the objectivists here: Everything I have heard that measure goods here on the forum is neutral and very well behaved. So at the moment, I use the measurements to guarantee that the "ingredients" and the "kitchen" are in order and that the "flavors" are balanced, but there are still areas not covered.

Amps that "add flavor" to a recording need to be pushing serious distortion to make any audible difference and even most tube amps don't get there in normal use.

Consider: a good conventional bookshelf speaker sitting at typical 75–85 dB listening levels will throw around 3% intermodulation distortion(IMD) on bassy tracks. IMD is the most audible distortion type after clipping, and at 3%, it's already masking whatever subtle character your amp might be contributing. You're not hearing your amp through that you're hearing your speakers or the differences you hear are products of cognitive biases.

The only scenario where a tube amp's coloration might genuinely poke through is with a full-range system running hefty subwoofers that offload the bass from the main drivers, dropping their distortion low enough for amplifier artifacts to become relevant. And even then, we're talking about the most aggressively colored tube designs.

So when people claim they hear differences between solid-state amps... no. What they're hearing is gain mismatch. Level-match them properly and 99.99% of solid-state amps are audibly transparent through any real-world speaker. Your brain is incredibly good at constructing narratives from expectation bias, and the audiophile world is basically a shrine to that phenomenon.

you do not believe me: https://www.klippel.de/listeningtest/?page=test_setup

test how good your distortion hearing ability is. Even the worst tube amps have about -50db distortion.

Also, unfortunately Youtube videos are not a good source to test and compare equipment. Youtube compression algorithm works in mysterious ways.

 

[RandomEar]

I will try to enter this debate as gentle as I can, with a point in favor of the subjectivists. Please bear with me as I walk on a shady ground.

As far as every measurement done, it is flawless and outperforms every human organ. Make no mistake about that.

To this day, I have never been able to discern between dacs (the delta-sigma ones) and I suspect that I probably never will.
On the other hand, every speaker I try is unique, even if they measure quite similar. There are of course degrees to these differences.

So far I think most of us can agree, but where I learned my lesson was with amplifiers. The simplest block in the signal chain. The PA5 II vs B100. I simply though that there would be no differences what so ever, but it was apparent immediately that there was a difference. I got schooled and did my own measurements, but they reveled no differences. So I chalked it up to my own bias and moved on with my life.

B100 died, so I had to go back to my PA5 II, and holy hell, no one can tell me that there is no difference. As I stand, I think you have to deaf to not actually hear the difference.

This puts me in an a conundrum: I have seen the measurements, done my own measurements, convinced myself that my brain is a bias machine, but here I stand.
Personally, I think that for amps there probably is a measurement that isn't done that would reveal this. I tried to do some research and perhaps there is something to the lack of bandwidth in a class-d amplifier that does it?

This video here kind of catches what I hear between a PA5 II and B100s:

With all that said and done. In favor of the objectivists here: Everything I have heard that measure goods here on the forum is neutral and very well behaved. So at the moment, I use the measurements to guarantee that the "ingredients" and the "kitchen" are in order and that the "flavors" are balanced, but there are still areas not covered.

You're making the same mistake as all the other subjectivists: "I think I can hear a difference (in my flawed testing), therefore there has to be one!"

You didn't test your amps:

• level-matched (<0.2 dB)
• blind or ideally double blind
• with immediate switching (<3 s)

So your results are biased and "tainted" in multiple ways. It's always the same discussion, it's always the same outcome: If you test wrong, the results of your test will be invalid. If you try to interpret invalid results, you can argue for or against any outcome you want to see.

 

[Jazigo]

You're making the same mistake as all the other subjectivists: "I think I can hear a difference (in my flawed testing), therefore there has to be one!"

You didn't test your amps:

• level-matched (<0.2 dB)
• blind or ideally double blind
• with immediate switching (<3 s)

So your results are biased and "tainted" in multiple ways. It's always the same discussion, it's always the same outcome: If you test wrong, the results of your test will be invalid. If you try to interpret invalid results, you can argue for or against any outcome you want to see.

You can find my limited measurements here: https://www.audiosciencereview.com/...onse-enough-to-determine-sound-quality.68183/

"I can hear a difference, therefore there has to be one!"

Don't get me wrong: I love audiosciencereview and it has been a tremendous help in so many ways that I can't thank everyone enough, but we should definitely not consider ourself done. There are more thing at play that we should be able to put values on.

 

[RandomEar]

You can find my limited measurements here: https://www.audiosciencereview.com/...onse-enough-to-determine-sound-quality.68183/

"I can hear a difference, therefore there has to be one!"

Again: You think you can. This is going to be a circular argument again...

Your own words concerning the level-matched comparison after DeltaWave:

I am never able to get a clean run, best have been 8/10, but in the end, I am never consistent with it.

So, blind and level matched there does not seem to be a statistically significant difference, right?

I don't know how many times you tried, but if you repeat the ABX often enough, you will get individual runs with 10/10. You can, however, combine the results of all test runs and recompute the statistics to get a more reliable result.

Don't get me wrong: I love audiosciencereview and it has been a tremendous help in so many ways that I can't thank everyone enough, but we should definitely not consider ourself done. There are more thing at play that we should be able to put values on.

I don't know how you captured the data from your linked thread. There seem to be small oscillations in the upper treble (even in the B100 vs B100 case) which look like measurement artifacts to me at first glance. But without knowing what and how this was captured, we can't really analyze them any further.

Maybe it makes sense to measure the phase curve for each amp, just to check that out.

 

[ahofer]

I don't know how many times you tried, but if you repeat the ABX often enough, you will get individual runs with 10/10. You can, however, combine the results of all test runs and recompute the statistics to get a more reliable result.

Reminds of when Stereophile got excited when one person in audience of 50 or so got the blind test questions all right. Time for a classic:

 

[rdenney]

The get out clause So basically what you are saying is there "Is no difference". Well that was very helpful. Thanks.

No, it's just trying the easiest and most likely explanations first.

Have you measured the audio output to see if both are at exactly the same voltage? Even 0.1 dB difference (23 millivolts difference in a 2-V signal) is enough to change perceptions. This is done with a 4.5-digit (minimum) voltmeter that has at least 100K of bandwidth if the signals you are measuring are at higher audio frequencies where you suspect a problem.

Have you tested the alternative inputs to your switching preamp or control box? If it's an old preamp (like all of mine), a dirty switch contact is not unlikely. If you are plugging directly into a power amp, and switching manually, the time required to move the cables is long enough to forget subtleties in the sound.

Have you tested the two devices using simple test signals and measured outputs? All DACs have an analog audio output section, and a fault in that can cause added distortion at high frequencies. This will affect not only steady-state tones but also transients with rise times as steep as those high frequency sine waves, or the occurrences of those frequencies in complex signals.

Philosophically, there are two assumptions you can make: A. Established and authoritative measurements and their meanings have shown no likelihood of perceivable differences, and so the explanation is probably one of the below three issues, or B. untested perceptions trump those established and authoritative measurements. I think you know where the locus of belief will be on ASR. But I think you are obligated to eliminate A rigorously before you can assert B, or even not be wasting your time trying to measure B.

If you are willing to challenge B as a starting point, as most here would be, then the three possibilities are (I'm assuming exactly matched outputs):

1. Malfunction of the device under test,

2. Perceptions that do not stand up to proper subjective testing ("blind" testing) that eliminates bias, or

3. A fault in the surrounding system that has gone undetected.

Those three must be eliminated, it seems to me, before one can reject Assumption A.

If you do carefully and appropriately eliminate those possibilities, then I think I might digitally record a random signal generator so that you have a repeatable sample, run them through the devices under test into test equipment to compare them. Or, simply find the frequency of the effect you perceive, and test the devices at a range of specific tones in that frequency vicinity.

Rick "finds those sorts of faults frequently enough" Denney

 

[rdenney]

...
On the other hand, every speaker I try is unique, even if they measure quite similar. There are of course degrees to these differences.

..

Measurements of speakers may be consistent but they do not attempt to measure the interaction of the speaker with the listening environment. All listening rooms are aggressive in their interaction with speakers, compared to the comparatively negligible interaction of surrounding electronics to line-level sources like DACs.

Rick "noting the difference between gross effects and fine effects" Denney

 

[Dan Lavry]

Music is about air vibrations, air pressure variations. If testing two devices extensively measure the same, should they not sound the same?

I can’t solve the mystery. But I can see some possible explanations. I would suspect that the device under test, hooked to a test system operates under different conditions then it does during listening. In other words, the problem has to do with the device interface with the system.

Here are a couple of possible examples:

A lot of analog output have some resistor in series. It is there for a reason (I will not go into it here), but say for example the value is 100 Ohm. Say your feed the signal into a test gear with 10K ohm input resistance. The voltage “loss” is very small, 20*log(10000/10100)= -0.086427dB attenuation.

Now hook the same device to and amplifier with input resistance of 600 Ohm. the level loss is now 20*log(600/700)= -1.338dB. The result for 5K load ohm is 0.172dB…

True, 600 is low but the DA’s should still operate well. I am not even touching the impact of the load (such as power amps or mixers) on distortions across the frequency range. The basic signal level depends on the DA output resistance and the load. So different DA's with different output resistance yield different levels.

Another possibility has to do with how a device handles jitter. The test system digital signal has very low jitter, so the DA does not show how the device would react to A signal from a jittery digital audio player. Different DA’s jitter rejection can be really good, or very poor… (I Don’t measure jitter with audio tester. I use different gear for jitter measurements).

I don’t claim to have the answers, but I can come up with even more possibilities. It is easy to speculate. But I have no reason to doubt reports from really good and experienced ears (better than my old ears with 8KHz bandwidth hearing aids).

 

[j_j]

The get out clause So basically what you are saying is there "Is no difference". Well that was very helpful. Thanks.

Excuses much? No, nobody said "is no difference", rather that there is no PROVEN difference. Without a good DBT including both positive and negative controls, nobody knows.

It is, however, quite true that at some (well below threshold) level there must be difference, as physics works regardless of any claim otherwise. The charge on the electronic is well established.

What is very much not established is "and how is this audible".

Do us a favor and show us that, ok?

 

[j_j]

Here are a couple of possible examples:

A lot of analog output have some resistor in series. It is there for a reason (I will not go into it here), but say for example the value is 100 Ohm. Say your feed the signal into a test gear with 10K ohm input resistance. The voltage “loss” is very small, 20*log(10000/10100)= -0.086427dB attenuation.

Now hook the same device to and amplifier with input resistance of 600 Ohm. the level loss is now 20*log(600/700)= -1.338dB. The result for 5K load ohm is 0.172dB…

True, 600 is low but the DA’s should still operate well. I am not even touching the impact of the load (such as power amps or mixers) on distortions across the frequency range. The basic signal level depends on the DA output resistance and the load. So different DA's with different output resistance yield different levels.

Another possibility has to do with how a device handles jitter. The test system digital signal has very low jitter, so the DA does not show how the device would react to A signal from a jittery digital audio player. Different DA’s jitter rejection can be really good, or very poor… (I Don’t measure jitter with audio tester. I use different gear for jitter measurements).

I don’t claim to have the answers, but I can come up with even more possibilities. It is easy to speculate. But I have no reason to doubt reports from really good and experienced ears (better than my old ears with 8KHz bandwidth hearing aids).

There are also interactions with cable impedance, input (non-resistive) impedance, and a lot of other things.

The irony, of course, is that when you get to be accurate enough, NOTHING measures exactly the same. Physics requires this. On the other hand there is no evidence of that level of matching mattering.

 

[Chrispy]

Known differences are the harshness in the D90SE compared to the X26III.

Like I said, you have some measurements or this is just your personal opinion?

 

[Dan Lavry]

I can’t help much, but a discrepancy between measurements and double-blind listening may be due to the difference operating a device in the laboratory (hooked to test gear) and at different location, hooked to audio gear. This is just a speculation on my part.

I went to few double-blind tests (carrying prototypes). I did not carry an Audio Precision system to measure and probe at different points of the signal path.

I will never forget the first time. It was to compare three early sample rate converters. There were three mastering engineers for the listening to various music. One of them was from Japan, his English was really limited. After the test he said: “Uni one OK. Uni two fah. Uni 3 good”.

That was many years ago, but I was already wondering then about the studio setup compared to the laboratory. But we live in the real world. As a designer and manufacture I am interested in doing the best I can on my side. I can’t guarantee that all the other “devices” will behave within specifications limits (such as AES/EBU or SPDIF). All it takes is one poor performer in the signal path…

But again, this is just a speculation on my part. I can't think of another cause if the DBT is done correctly.

 

[voodooless]

I went to few double-blind tests

…

After the test he said: “Uni one OK. Uni two fah. Uni 3 good”.

That would be very strange, because that’s not the type of conclusion that such a test gives.

 

[antcollinet]

So basically what you are saying is there "Is no difference". Well that was very helpful. Thanks.

No, they are saying you don't know there is a difference without rigorously controlled listening comparisons. And all the evidence states there is no difference to be heard, so in the absence of sufficiently controlled comparisons, it is vanishingly unlikely that what you are hearing is coming from the DAC.

Known differences are the harshness in the D90SE compared to the X26III.

No, that is just an oft-repeated fallacy. And its often-repeated nature may well be what is causing you to hear it via your naturally human perceptive biases. The brain part of our auditory system is plastic. It can change what you hear for all sorts of reasons.

The harshness is obviously worse at higher volumes.

That might result from pushing your amp into clipping or overloading the speakers. This is one reason why comparison against another DAC (that doesn't sound like this) and accurately level-matched with a volt meter (microphone/decibel meter
is not accurate enough) to 0.1% is needed. The other reason is the tell that even tiny volume differences can create in the brain.

The WHY bit is very intriguing. Hence the original post to a Science forum.

There is no point considering the why until there is a properly determined what. Otherwise, you are just chasing your tail.

Just because there isn't a scientific method of measuring something, it doesn't mean it doesn't count or can be ignored. Cheers.

(my bold)

Ah, that old chestnut.

Audio is one of the oldest and simplest applications of electronics. We have been doing it for almost exactly 150 years. There is practically nothing we don't know about the physics of it, and we have a good understanding of the related human auditory capabilities.

Audio signals are amongst the simplest of physical phenomena to measure. There is only amplitude, frequency and phase, distortion and noise. And we have measurement equipment able to measure those to a sensitivity way beyond that of our ears. So far, not one person has demonstrated that there is an audible difference that can arise when the established device measurements show only inaudible inaccuracies.

And this despite all the money and vested interest in the industry to prove that their products can be scientifically proven to have the audio magic they claim in their marketing. Nothing, nada, crickets.

 

[Audionaut]

somebody or other was annoyed because it was "too good".

Why that?

 

[Audionaut]

What is the proper terminology these days? (I thought that high end means high performance).

I understood you as you meant it, but I have noticed that the term on ASR has different connotations depending on cultural affiliation and nationality.
Most Germans for example understand high-end in the audio sector to mean what is technically possible, regardless of price or appearance.
In the US, it tends to refer to overpriced audio junk, if I understand correctly (which is not always the case).

 

[Audionaut]

until someone comes along with scientific explanation and test results to show the difference. At this moment in time I don't think we are quite there yet

This is the case in the absolute fringes of science, but the audio story has been fully told and researched on the electronic side.
This is not about quantum mechanics, evidence for string theory, or the fringes of universal origins, but rather a relatively narrow window in the human realm of perception, for which the now much more powerful electronics are being used.
This has been researched in its entirety, just like the laws of leverage and pulleys. No one is going to come around the corner and shout, "Eureka, I've found the philosopher's stone, we all need to radically rethink everything. Paradigm shift."
The bottleneck is human perception, not electronics. Most of the people who write in audio forums talk and argue heatedly about differences in electronics that are imperceptible to humans, but they don't know their own hearing profiles and perception limits. As a rule, these are much more limited among the average members of audio forums than their ability to philosophize about differences that are not measurably perceptible.
The placement of a loudspeaker in a room for example has a much greater impact on the perceived sound than the difference between two modern DACs could ever have.

 

[j_j]

Why that?

It was better than necessary. Some folks did not like that. You may ask. I do not know why.

 

[Audionaut]

It was better than necessary. Some folks did not like that. You may ask. I do not know why.

For me, that's a Schrödinger answer; I understand it and at the same time I don't understand it.