Why DA/AD diff tests not getting the attention it deserves in measurements

[dasdoing]

might be of intrest: https://www.zmix.net/Converter_Test/index.htm

 

[Grooved]

My first guess might be a little more hum on unbalanced. With nulls of -89 db and lower it takes very little to worsen a null. As Paul said it should show up in FR or maybe phase (though I don't know why there would be a phase difference in balanced vs unbalanced).

So what are the differences if you null these two file captures against each other?

Here are the null results of both files, 1 is with Deltawave basic settings, and 2 with advanced settings enabled like bellow

Basic settings:

Advanced settings:

 

[pkane]

Here are the null results of both files, 1 is with Deltawave basic settings, and 2 with advanced settings enabled like bellow
View attachment 232790

Basic settings:
View attachment 232791
View attachment 232792

Advanced settings:
View attachment 232793
View attachment 232794

Non-linear EQ is an advanced function, as I mentioned. It is sensitive to noise as it's performing a deconvolution of the original signal based on estimated impulse response. If the impulse response is noisy (the errors in spectrum/phase are random), deconvolution will not produce a good result. To help average out the noise, you may want to use smaller FFT size, for one. But, there's no guarantee that deconvolution will help in all cases -- it depends on the data, the original spectra, and the amount of noise. Check phase and spectrum differences with non-linear eq and without to see what effect it had on both.

 

[Grooved]

Another thing I saw is that the main parts what gives lower difference results are when high frequencies are at higher levels.
We looked at DC, power supplies,... but not high frequencies.

This what happens at 1'43" in the "original" file (quickly captured from Foobar just to show): there was a jump (cymbals) just before, in grey in the picture
It also happens two other times later

and just looking at two captures I made with 828MK2 and iD22 to check with another one, it's exactly where the difference has the worst result, for both devices and I think for most of all devices:

828MK2:

iD22:

So are high frequencies the main problem?

 

[Blumlein 88]

Another thing I saw is that the main parts what gives lower difference results are when high frequencies are at higher levels.
We looked at DC, power supplies,... but not high frequencies.

This what happens at 1'43" in the "original" file (quickly captured from Foobar just to show): there was a jump (cymbals) just before, in grey in the picture
It also happens two other times later
View attachment 232811

and just looking at two captures I made with 828MK2 and iD22 to check with another one, it's exactly where the difference has the worst result, for both devices and I think for most of all devices:

828MK2:
View attachment 232812

iD22:
View attachment 232813

So are high frequencies the main problem?

Could be, often differences in filter used cause differences in high frequencies. Especially with phase. If you don't use the advanced features, look at the phase and see if they diverge at the higher frequencies. Two devices could match very well, and even have nearly the same frequency response, but a phase response idfference can make the null less good. The advanced features of course compensate for that.

 

[Grooved]

Could be, often differences in filter used cause differences in high frequencies. Especially with phase. If you don't use the advanced features, look at the phase and see if they diverge at the higher frequencies. Two devices could match very well, and even have nearly the same frequency response, but a phase response idfference can make the null less good. The advanced features of course compensate for that.

Thanks, I thought about the filter at first, and I will check phase.

 

[Nickerz]

Why do you care about combined performance of these two subsystems? If you are producing music, then the ADC is what matters, not the DAC. If you are playing music, then the DAC matters and not the ADC.

As noted, it is *extremely* non-trivial to create proper null/difference tests. I have spent days at times trying to get what I know to be correct results and failing.

Even if you got correct null results, interpretation is impossible. You don't know what is or is not audible since you have taken out the masking signal in the source.

Because there are a lot of usecases for going out of the DAC and into the ADC. For instance "hardware inserts", sending audio for destructive processing as well as a final print through hardware 2bus (this is functionally the same as a hardware insert as well). These are all essentually DA\AD loops and it's common to happen a couple times for every project. And the quality of this for the "final print" is very important if using an analog mix bus.

 

[DonH56]

Because there are a lot of usecases for going out of the DAC and into the ADC. For instance "hardware inserts", sending audio for destructive processing as well as a final print through hardware 2bus (this is functionally the same as a hardware insert as well). These are all essentually DA\AD loops and it's common to happen a couple times for every project. And the quality of this for the "final print" is very important if using an analog mix bus.

The focus of most (almost all) of @amirm 's tests target consumer, not studio, applications so there is usually none to maybe one or two back-to-back AD/DA conversions in the signal path. Even for studio work, I agree with Amir that other processing along the way will dwarf changes introduced by the data converters. And doing a null test on an unknown chain (in terms of really knowing the transfer function) is impractical if you are looking for 100 dB nulling -- too many other error sources creep in that are virtually impossible to quantify and compensate.

 

[Nickerz]

The focus of most (almost all) of @amirm 's tests target consumer, not studio, applications so there is usually none to maybe one or two back-to-back AD/DA conversions in the signal path. Even for studio work, I agree with Amir that other processing along the way will dwarf changes introduced by the data converters. And doing a null test on an unknown chain (in terms of really knowing the transfer function) is impractical if you are looking for 100 dB nulling -- too many other error sources creep in that are virtually impossible to quantify and compensate.

Right, that's why a loopback test is actually very valuable. You have one variable, the cable. It will tell you that if you're processing your own audio how well it performs in relation to other devices, and no surprise, the ones that are considered to sound good in studios tend to rise to the top. Not all, but the ones that have the best word of mouth. Somewhat correlated to price but not the only or even leading indicator.

The intersection of the AD DA list and these stats published here are the cream of the crop.

 

[DonH56]

Right, that's why a loopback test is actually very valuable. You have one variable, the cable. It will tell you that if you're processing your own audio how well it performs in relation to other devices, and no surprise, the ones that are considered to sound good in studios tend to rise to the top. Not all, but the ones that have the best word of mouth. Somewhat correlated to price but not the only or even leading indicator.

The intersection of the AD DA list and these stats published here are the cream of the crop.

For a null test such as discussed previously, you have the transfer function of the ADC and DAC, with all the various filters (anti-aliasing, anti-imaging), so the cable is a minor variable. Look through some of the earlier posts for more detail on the problems with performing a null test on even an apparently simple system. You can measure the performance easily enough, and compare to the first signal in the chain, but performing a null test is a whole 'nuther beast.

 

[Nickerz]

After spending a lot of time thinking about this and going through a lot of different ways to pick this apart, I think the DeltaWave approach is wrong based on anecdotal evidence. I think the forum should spend time brainstorming ways to measure ADC performance in ways not previously thought of to try and derive a new approach. I know many or most members won't be open to this mode of thinking because they are influenced by ways of thinking that are non-musical. But suffice to say, I have dove into this enough to reject DeltaWave entirely and use the GearSpace loopbacks simply as a junk spotter, and not as a way to determine the best ADC conversion. I have fallen back on the wisdom of crowds and will wait for people to find a way to quantify that which doesn't seem easily quantified at this point.

I believe this view is rational, but I don't care to argue it too much harder than I have already and wish you guys all the best in the persuit towards your perfection.

Cheers.

 

[pkane]

I think the DeltaWave approach is wrong based on anecdotal evidence.
...
I believe this view is rational

Really? Your rational view is based on anecdotal evidence?

How is it that you reject DeltaWave and yet use GearSpace loopback results when both compute the same value? In fact, most results agree to a tiny fraction of a dB? DeltaWave then goes on to compute about two dozen other results, including phase, audibility metrics, linearity, even the impulse response and transfer function. When you say it's wrong, what do you mean, and what anecdotal evidence do you have to support this?

 

[Blumlein 88]

After spending a lot of time thinking about this and going through a lot of different ways to pick this apart, I think the DeltaWave approach is wrong based on anecdotal evidence. I think the forum should spend time brainstorming ways to measure ADC performance in ways not previously thought of to try and derive a new approach. I know many or most members won't be open to this mode of thinking because they are influenced by ways of thinking that are non-musical. But suffice to say, I have dove into this enough to reject DeltaWave entirely and use the GearSpace loopbacks simply as a junk spotter, and not as a way to determine the best ADC conversion. I have fallen back on the wisdom of crowds and will wait for people to find a way to quantify that which doesn't seem easily quantified at this point.

I believe this view is rational, but I don't care to argue it too much harder than I have already and wish you guys all the best in the persuit towards your perfection.

Cheers.

What is your anecdotal evidence and what other ways of measuring ADCs do you think would be useful?

 

[Nickerz]

Really? Your rational view is based on anecdotal evidence?

How is it that you reject DeltaWave and yet use GearSpace loopback results when both compute the same value? In fact, most results agree to a tiny fraction of a dB? DeltaWave then goes on to compute about two dozen other results, including phase, audibility metrics, linearity, even the impulse response and transfer function. When you say it's wrong, what do you mean, and what anecdotal evidence do you have to support this?

Basically it came down to this decision tree.

Mastering engineer with 20k credits and a whole shelf full of gold statues is picking a mediocre converter or the measurement can only be a way to find bad converters. Then instantly think "oh... Deltawave must be right and GS is wrong" but the converter in question ranked even lower using DW. So after spending a lot of time, and mind you this is ADC, not DAC, my personal view on the matter is the measurement will only reveal bad conversions. And the rest is subjective and we don't yet have a good measurement to determine this.

Some of the people that are way deeper into this than me think it's a time domain issue and that good converters are more accurately representing time domain or some other measurement we don't have access to yet.

I think it's easy to claim he has been bamboozled, but harder to admit that this measurement is simply not as important as we think. More of one part of the picture of one we don't yet understand that well. Billions of dollars and hundreds or thousands of scientists in more pressing issues are wrong all the time and I don't think it's controversial to claim we may be off here.

Basically I trust the engineer's ear more than I trust a number to do anything more than show it is good in one dimension of measurement.

 

[Jim Taylor]

Basically I trust the engineer's ear more than I trust a number to do anything more than show it is good in one dimension of measurement.

This is what you're really saying.

We've got gene splicing inside cells going on every day, cell phone coverage of the whole planet, MRI that detects every abnormality in a person's head, radar that tells us the location and intensity of tornados before they hit, GPS that delivers artillery shells to within the area of a car tire, satellites that give us pictures of Mars and the outer planets, all by virtue of measurements done by electronic instruments, but when it comes to audio, measurements done by electronic instruments are inferior to someone's ear, right?

Have I got that right?

Jim

 

[Killingbeans]

Some of the people that are way deeper into this than me think it's a time domain issue and that good converters are more accurately representing time domain or some other measurement we don't have access to yet.

That seems to be the default hand-waving when measurements indicate that people are simply imagining things. "Time domain" this and "time domain" that.

It's as if they believe Fourier transforms only work once in a while if you're lucky.

I see it as a "what if you're wrong?" kind of argument. I don't really care if I am. If something has an odd chance of being relevant, I'd rather spend my time on the things that have proven to be effective, and leave the shifting of odds to the people who do serious research. Otherwise the risk of walking around in circles is just way to big.

 

[Blumlein 88]

Basically I trust the engineer's ear more than I trust a number to do anything more than show it is good in one dimension of measurement.

This indicates a basic misunderstanding of what Deltawave or a null test is showing. It shows how good something is in all possible ways of measurement at once.

 

[pkane]

Basically it came down to this decision tree.

Mastering engineer with 20k credits and a whole shelf full of gold statues is picking a mediocre converter or the measurement can only be a way to find bad converters. Then instantly think "oh... Deltawave must be right and GS is wrong" but the converter in question ranked even lower using DW. So after spending a lot of time, and mind you this is ADC, not DAC, my personal view on the matter is the measurement will only reveal bad conversions. And the rest is subjective and we don't yet have a good measurement to determine this.

Some of the people that are way deeper into this than me think it's a time domain issue and that good converters are more accurately representing time domain or some other measurement we don't have access to yet.

I think it's easy to claim he has been bamboozled, but harder to admit that this measurement is simply not as important as we think. More of one part of the picture of one we don't yet understand that well. Billions of dollars and hundreds or thousands of scientists in more pressing issues are wrong all the time and I don't think it's controversial to claim we may be off here.

Basically I trust the engineer's ear more than I trust a number to do anything more than show it is good in one dimension of measurement.

Sorry, can't help you there. I don't know how to measure a subjective preference of some engineer from a loopback recording. Maybe we need to loop it through his brain, first?

 

[DonH56]

I am struggling to see how DW works if not in the time domain...

IEEE Standards for ADC and DAC testing are pretty thorough, and I am sure there are others. By engineers, though, so I guess that obviates that 40 or 50 years of research and development. If only I had known before participating in their development that all I really needed to do was to listen to them.

Really tired of the "I don't understand what you are saying, but you must be wrong" argument.

 

[pkane]

I am struggling to see how DW works if not in the time domain...

Indeed...