Why DA/AD diff tests not getting the attention it deserves in measurements

[Waxx]

Much gear in audio production is used because they colour, even some ad/da devices are used like that. But that does not mean they are technically superior, more the opposite. The reason why neve and api preamps are classics is because of that, they colour in a specific way the sound, and that colour is something many like.

And mastering engineers also use a lot of those tricks to polish the production. That's why they mostly have some colouring devices (eq's, compressors, ...) next to the swiss knive precision devices (analog or digital) in their setup. And no, they are not immune to snake oil.

The tests Amir does are scientific and measure how precise the conversion is, not how it colours or would be fit for mixing or mastering. If it's something you need is something you need to decide. And no, not everybody wants super clean sound (i don't), but even for people like us his measurents tell a lot that subjective reviews or bloated tech reviews like the ad/da diff tests are giving.

 

[pma]

But people with unlimited money absolutely have a preference

Yes they have a preference, but same time not much abilities to find a difference in a DBT, so they avoid it and call it flawed.

 

[Nickerz]

Much gear in audio production is used because they colour, even some ad/da devices are used like that. But that does not mean they are technically superior, more the opposite. The reason why neve and api preamps are classics is because of that, they colour in a specific way the sound, and that colour is something many like.

And mastering engineers also use a lot of those tricks to polish the production. That's why they mostly have some colouring devices (eq's, compressors, ...) next to the swiss knive precision devices (analog or digital) in their setup. And no, they are not immune to snake oil.

The tests Amir does are scientific and measure how precise the conversion is, not how it colours or would be fit for mixing or mastering. If it's something you need is something you need to decide. And no, not everybody wants super clean sound (i don't), but even for people like us his measurents tell a lot that subjective reviews or bloated tech reviews like the ad/da diff tests are giving.

I agree, but in this case it was his "straight line" converter. He was clear he used other converters for color\texture.

 

[Killingbeans]

Again, I'm not entirely rejecting the measurement, I think it is a valid way to root out bad converters, but you'd have to come up with a more compelling reason to me why very talented individuals with great ears are buying "mediocre" gear for huge cash other than they're gullible and there's a cabal of companies selling hugely overpriced snake oil boxes because a measurement everyone says people can't use to a\b test themselves is the way.

That's the problem. The real reason is not compelling at all.

You can't train or will your way out of cognitive bias, and that's just a sad fact of life. It's a particularly hard pill to swallow when your livelihood is dependent on the reputation of your hearing skills as a professional tool. I have no doubt that these individuals are extremely talented, but it doesn't help them to avoid this pestering side effect of human biology. On the contrary.

 

[Blumlein 88]

That's the problem. The real reason is not compelling at all.

You can't train or will your way out of cognitive bias, and that's just a sad fact of life. It's a particularly hard pill to swallow when your livelihood is dependent on the reputation of your hearing skills as a professional tool. I have no doubt that these individuals are extremely talented, but it doesn't help them to avoid this pestering side effect of human biology. On the contrary.

The only people worse than music recording pros about their special hearing ability and why they aren't biased seem to be those who were sonar operators. It is obvious why they would have this opinion. Yet despite experience, training and real skill, in the end they cannot escape common human bias about things.

 

[Nickerz]

That's the problem. The real reason is not compelling at all.

You can't train or will your way out of cognitive bias, and that's just a sad fact of life. It's a particularly hard pill to swallow when your livelihood is dependent on the reputation of your hearing skills as a professional tool. I have no doubt that these individuals are extremely talented, but it doesn't help them to avoid this pestering side effect of human biology. On the contrary.

Bias is not absolute and with certainty beyond competence. Bias is a tendency. In the same way I can drive drunk a lot before I get into an accident. People that think some bias is an absolute certainty to making the wrong choice also believe that a drunk driver is almost certainly going to crash on the way home. Despite a drunk person being 600 times more likely to crash on that trip, the vast majority will make it home fine.

Being strongly influenced away from the path of truth isn't a certain trip to the wrong choice, just a concern on your path to truth.

 

[Killingbeans]

I never said it was an absolute. But the tendency is reinforced every time there's a lack of consequence.

The drunk driver gets home safe and takes it as a sign of his driving skills being so extraordinaire that he will never get into an accident. And so he repeats the moronic action again and again.

The sound engineer also has no reason to question his cognition as long as it's telling him/her what he/she wants to hear.

You asked why so called "talented individuals" could possibly be susceptible to snake oil, and my answer is: Because they are humans, just like the rest of us.

 

[Geert]

The only people worse than music recording pros about their special hearing ability and why they aren't biased seem to be those who were sonar operators.

I am a music recording pro. I'm successful at what I do because already 30 years ago I recognised my hearing capabilities are no match for measuring equipment. (Nevertheless, hearing is a skill you can develop to a certain extent).

 

[pkane]

Being strongly influenced away from the path of truth isn't a certain trip to the wrong choice, just a concern on your path to truth.

That's certainly not a fact. A strong, unquestioned influence can easily lead one astray, and sometimes for a lifetime. The "support" provided by the audiophile press and many hi-fi manufacturers, the made up creative marketing and complete nonsense designs just reinforce the natural, lay person's inclination to trust your own senses.

Science, on the other hand, is clear that human senses are not designed for measurement and detection of minute differences. They lack the resolution, the bandwidth, and the detection ability of specialized instruments. By using more sensitive instruments, proper research can then find the correlation between measurements and our hearing abilities, hearing thresholds, and the ability to recognize differences. This is how we know a lot about our hearing, how we process sound, and what natural limits there are on this processing.

Measurements in isolation are mostly meaningless, but in relation to knowing our hearing thresholds, bandwidth, resolution measurement can provide a much better test for a good sound recording/reproduction quality than someone's random preferences.

 

[Cbdb2]

People in fact do not wish to participate in ABX tests. If you prepare one, which is quite time consuming due to proper recording, time aligning and level matching, almost no one will come with the ABX report. People like to chat, mostly. Not to work, not to learn something.

And they don't like the proof that there hearing is average when audiophool sites are full of people who pretend to have super hearing.

 

[Blumlein 88]

I am a music recording pro. I'm successful at what I do because already 30 years ago I recognised my hearing capabilities are no match for measuring equipment. (Nevertheless, hearing is a skill you can develop to a certain extent).

I did not intend to paint with such a broad brush as to mean all recording pros. It is a common idea to point to pros who mimic or at least concur with lots of audiophile foolishness. Many pros don't do such a thing, but audiophiles don't want to talk about them. My limited exposure to such people is that at least 50% don't go along with the foolishness. Even some who don't have all that much technical understanding, but just have their experience and skill using gear.

 

[Grooved]

Hi, I had a hard time trying to understand what @Nickerz is exactly talking about, and specifically this supposed device that has different results between DeltaWave and GS test, as they all get the same results in both tests when DeltaWave is used without correction settings turned ON (which seems logical, because when the signal is back in the DAW, it doesn't have been corrected by any software between the ADC device and the DAW).

But, even if I'm not sure it's the kind of things he's talking about, I found one thing that might match the kind of question is having right now:

I did a test with the Tone2 Pro and Cosmos ADC (no sync between both, so I ran it several times to check it was not having a huge impact).
In Multitone, with any filter, I get the same THD+N value, and switching between filters should gives a very small change regarding the 20-20k frequencies range, which made me expect a very small change in DeltaWave results too.

But after trying all filters, I got a difference in DeltaWave ranging from -46 to -55dB
And I have to admit that at first, I hadn't expected this 9dB change in the results, since it's supposed to measure from 20Hz to 20KHz, where the different filters are at worst slightly impacting both end of this frequencies range?
I know it's more linked to phase than frequencies, and also that DC filter, if turned OFF or ON on the ADI 2-PRO, can also change the results by a huge margin, while one could think that it's not supposed to have any impact on 20Hz and higher, so the first questions look like :
Does it means that even if the DC filter is working out of the audible range, it impacts it?
or
Does it comes from the software measuring from 0 to 22.05kHz instead of 20-20k?
or
Does it comes from phase change between the original file and the recorded one?
or
...
And it leaves you at first with a taste that the results are not explaining you what is clearly making the difference.

So, @Nickerz are you talking about this kind of measurement results that don't look like something you can clearly based immediately your opinion on?

 

[pkane]

Hi, I had a hard time trying to understand what @Nickerz is exactly talking about, and specifically this supposed device that has different results between DeltaWave and GS test, as they all get the same results in both tests when DeltaWave is used without correction settings turned ON (which seems logical, because when the signal is back in the DAW, it doesn't have been corrected by any software between the ADC device and the DAW).

But, even if I'm not sure it's the kind of things he's talking about, I found one thing that might match the kind of question is having right now:

I did a test with the Tone2 Pro and Cosmos ADC (no sync between both, so I ran it several times to check it was not having a huge impact).
In Multitone, with any filter, I get the same THD+N value, and switching between filters should gives a very small change regarding the 20-20k frequencies range, which made me expect a very small change in DeltaWave results too.

But after trying all filters, I got a difference in DeltaWave ranging from -46 to -55dB
And I have to admit that at first, I hadn't expected this 9dB change in the results, since it's supposed to measure from 20Hz to 20KHz, where the different filters are at worst slightly impacting both end of this frequencies range?
I know it's more linked to phase than frequencies, and also that DC filter, if turned OFF or ON on the ADI 2-PRO, can also change the results by a huge margin, while one could think that it's not supposed to have any impact on 20Hz and higher, so the first questions look like :
Does it means that even if the DC filter is working out of the audible range, it impacts it?
or
Does it comes from the software measuring from 0 to 22.05kHz instead of 20-20k?
or
Does it comes from phase change between the original file and the recorded one?
or
...
And it leaves you at first with a taste that the results are not explaining you what is clearly making the difference.

So, @Nickerz are you talking about this kind of measurement results that don't look like something you can clearly based immediately your opinion on?

All these questions can be answered by using the tools provided with DeltaWave. If you suspect that phase is the problem, try correcting for phase in non-linear EQ settings to see how much that affects the result. If you think it's the filter frequency response, try correcting for non-linear level and see how much the result changes. If you think it has to do with frequencies outside the audible range, use the A-weighted null RMS value or the PK Metric result.

All of these have been discussed many times in the GS thread, as well as, in this very thread, and in the DeltaWave and PKMetric discussions. The raw null RMS value, as reported in GS thread and computed as the basic, uncorrected result in DeltaWave is not a perception-based metric. It computes the result over the whole spectrum and doesn't account for variable group delay/phase differences, or for the audibility of the error. It is much more of an engineering metric that simply computes the raw error RMS value.

 

[Grooved]

All these questions can be answered by using the tools provided with DeltaWave. If you suspect that phase is the problem, try correcting for phase in non-linear EQ settings to see how much that affects the result. If you think it's the filter frequency response, try correcting for non-linear level and see how much the result changes. If you think it has to do with frequencies outside the audible range, use the A-weighted null RMS value or the PK Metric result.

All of these have been discussed many times in the GS thread, as well as, in this very thread, and in the DeltaWave and PKMetric discussions. The raw null RMS value, as reported in GS thread and computed as the basic, uncorrected result in DeltaWave is not a perception-based metric. It computes the result over the whole spectrum and doesn't account for variable group delay/phase differences, or for the audibility of the error. It is much more of an engineering metric that simply computes the raw error RMS value.

That's right @pkane , but what I was pointing at is that it might be easy during the first tests done with software like that (giving lots of results) to be a bit lost, and I wanted to know if this is part of what @Nickerz is talking about

 

[KSTR]

Does it comes from phase change between the original file and the recorded one?

Phase is the dominant factor, and it's most important in the low frequency range, hence the system highpass (DC-filter of the ADC even when the DAC is DC-couple) is the dominant contributor to the difference signal.

Two things are important here:

• The low-frequency roll-off of a DC-filter is always minimum phase, for practical reasons, even when it is a digital filter only.
• The spectrum of the test track is typical in that it mostly contains low frequency signals.

The point is that even when the magnitude of the filter is down only by, say, 0.001dB at, say 40Hz, this level difference will only be relevant (obtaining a rather deep null) when the filter is linear-phase... which it isn't. Even the exact same level wouldn't help as soon as the signals are not in phase anymore, and a few degrees are enough to completely spoil the null when energy in that frequency range is large, which it is.
That's why my RME ADI-2 Pro with completely removed DC-filter (digital filter switched off and DC-blocking caps shorted out) fared best whereas the RME Baybeface which has DC-blocking hardware filter at at pretty high frequency "failed" big time.

Only when the DC-filter is completely removed, passband frequency response ripple will be significant contributor to the residual but it is usually linear phase so no phase issues here. Same goes for the anti-imaging and anti-aliasing filters (which actually are responsible for the passband ripple). When those are selected to have sharp cutoff at high frequency in a linear phase fashion (often the standard choice for these filters) their impact is minimal. Then, and only then, real ill-effects like distortion and (huge amounts of) jitter would start to dominate the residual.

 

[pkane]

That's right @pkane , but what I was pointing at is that it might be easy during the first tests done with software like that (giving lots of results) to be a bit lost, and I wanted to know if this is part of what @Nickerz is talking about

That's what I wanted to find out, too, but it appears the whole discussion is based on "anecdotal evidence" of a mastering engineer picking an ADC that didn't score well in these measurements. As I said, I have no way to measure someone's preference from a loopback capture of his preferred interface

 

[Nickerz]

Hi, I had a hard time trying to understand what @Nickerz is exactly talking about, and specifically this supposed device that has different results between DeltaWave and GS test, as they all get the same results in both tests when DeltaWave is used without correction settings turned ON (which seems logical, because when the signal is back in the DAW, it doesn't have been corrected by any software between the ADC device and the DAW).

But, even if I'm not sure it's the kind of things he's talking about, I found one thing that might match the kind of question is having right now:

I did a test with the Tone2 Pro and Cosmos ADC (no sync between both, so I ran it several times to check it was not having a huge impact).
In Multitone, with any filter, I get the same THD+N value, and switching between filters should gives a very small change regarding the 20-20k frequencies range, which made me expect a very small change in DeltaWave results too.

But after trying all filters, I got a difference in DeltaWave ranging from -46 to -55dB
And I have to admit that at first, I hadn't expected this 9dB change in the results, since it's supposed to measure from 20Hz to 20KHz, where the different filters are at worst slightly impacting both end of this frequencies range?
I know it's more linked to phase than frequencies, and also that DC filter, if turned OFF or ON on the ADI 2-PRO, can also change the results by a huge margin, while one could think that it's not supposed to have any impact on 20Hz and higher, so the first questions look like :
Does it means that even if the DC filter is working out of the audible range, it impacts it?
or
Does it comes from the software measuring from 0 to 22.05kHz instead of 20-20k?
or
Does it comes from phase change between the original file and the recorded one?
or
...
And it leaves you at first with a taste that the results are not explaining you what is clearly making the difference.

So, @Nickerz are you talking about this kind of measurement results that don't look like something you can clearly based immediately your opinion on?

Not really so much to do with DeltaWave at all. I just don't think the measurement itself scales. I think it can be used to spot bad detectors but we have to fall back on ears after that.

 

[pkane]

Not really so much to do with DeltaWave at all. I just don't think the measurement itself scales. I think it can be used to spot bad detectors but we have to fall back on ears after that.

I'm fine with using ears to detect differences. The problem I have is that almost nobody is doing ears-only evaluation. Science and research is clear that such testing invalidates or, at least, casts a very strong suspicion, on any conclusions drawn from such testing.

 

[tuga]

most will use vintage compressors, sometimes without even using it to compress at all. at the end it's the same goal: create "saturation" (how they call "good distorsion")

But you know better?