A More Useful Way of Measuring the Performance of DACs?

[manisandher]

Would you be testing the original file vs a recording or compare 2 recordings and consider one 'reference' ?

Against the original file.

 

[danadam]

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then analyse these with DeltaWave. You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -70dB
- PK Metric = -50dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

The one time I happened to evaluate PK Metric, it hadn't worked that well for me. I didn't hear a difference at the point where it indicated -47 or -52 dBFS difference, but had no problem at one point with -56 dBFS difference.

 

[Scytales]

[...]

In any event, I wanted to explore what might be contributing to the differences I’m hearing. So, I set up the following 2 chains:

1. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> RME ADI-2 Pro BE @+13dBu -> USB -> RME Digicheck (on PC)
2. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> Prism AD124 @+12dBu -> BNC -> Tascam DA3000 (SD card)

[...]

I played three different 24/44.1 tracks and captured the analogue outputs at 24/44.1 in order to perform null tests with @pkane 's superb DeltaWave software.

1. 1kHz sine @0dBFS

[...]

Could DeltaWave provide any insights? Well, here are the results:

View attachment 245311

[...]

2. Gearspace ‘Original2’

This is the track used on the Gearspace loopback thread: https://gearspace.com/board/gear-sh...ing-ad-da-loops-means-audio-diffmaker-76.html. This is real music, and not a test signal.

Here are the DeltaWave results:

View attachment 245312

[...]

3. Diana Krall – Temptation

Here are the DeltaWave results:

View attachment 245316

Hi.

Thanks for your efforts.

However, I'm a bit lost by your first message.

I understand you made two different captures of the analogue output of each of your three DACs from the same test files with two different recording chains, each incorporating a different ADC. That's of no concern to me.

What bothered me is that I have not read anywhere which two captures have been compared with each other to obtain each and every null difference, whatever the metric or the recording chain.

On the Deltawave software website, it is explained that said software "allow detailed comparison of two different captures of the same audio signal".

OK, but what are the two different captures you have compared in Deltaware to produce the above cited charts ?

 

[danadam]

OK, but what are the two different captures your have compared in Deltaware to produce the above cited charts ?

As I understand this, these are the captures compared to the original file.

 

[manisandher]

Thanks for your effort.

Thanks.

What bothered me is that I have not read anywhere which two capture have been compared with each other to obtain each and every null difference, whatever the metric or the recording chain.

Each capture was compared to the original file that was fed into the DACs.

Mani

 

[manisandher]

On reflection, the question I posed earlier is pretty dumb. It's very unlikely that the PK Metric would ever be worse than the RMS Difference, and certainly not substantially worse.

 

[pkane]

A quick question for anyone who's looked into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then analyse these with DeltaWave. You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -70dB
- PK Metric = -50dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

Mani.

PKMetric starts with the RMS difference. In most DACs, the large RMS error is due to filters (reconstruction, anti-imaging and DC). Because these filters change FR around extremes, PKMetric will lower the error, not increase it by applying frequency-audibility threshold weights.

If PKMetric increases the error, this would be a very strange case where the device produces an error in the high audibility frequency range while decreasing it in the range where it’s normally not audible. Hard to imagine such a device, and can only happen if you use dBr units, since that’s relative to the signal level. dBFS PKMetric should not exceed RMS of the null signal.

Edit: I see you posted the same

 

[FINFET]

Using real music file diffs for dac measurements (instead of tone/multitones) could be very tricky. I once posted some similar question (but not with good measurements and experiments like op) and start to gradually understand the difficulty later about how filters and phase change could affect music file nulling. Some posts by members in that thread might interest you as well.

 

[pkane]

Archimago also started doing this a while ago (but not with Deltawave) but abandonned it.
There is also or Russian guy on this forum pushing his difference files. I believe just before Paul came with Deltawave. I am sure @pkane knows who this is. He already had quite a collection of data on phones, DACs, MP3 players.

When it is not an official standard way of testing I don't think Amir will adopt it.



The one with -50 is potentially audible different.
Whether it is RMS or PK metric it would warrant an investigation.
Would you be testing the original file vs a recording or compare 2 recordings and consider one 'reference' ?

Serge Smirnoff proposed a DF metric, which is similar to the RMS null and has no perception-based weights applied to the result. DeltaWave also computes DF Metric, so you have your choice of which metric to use

 

[manisandher]

Take Two...

For those of you who've had a chance to look into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then compare the captures vs. the original in DeltaWave (using only 'match gain' and 'subsample offset' settings). You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -60dB
- PK Metric = -60dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

@pkane , I'd love to hear which you'd choose, and why.

Mani.

 

[pkane]

Take Two...

For those of you who've had a chance to look into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then compare the captures vs. the original in DeltaWave (using only 'match gain' and 'subsample offset' settings). You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -60dB
- PK Metric = -60dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

@pkane , I'd love to hear which you'd choose, and why.

Mani.

If that’s the only thing I get to go by, then I’d pick the -70dB PKMetric. But a single number is never going to tell the whole story, so I’d prefer to look at the full PKMetric plot over the entire time range.

 

[Cote Dazur]

We listen to music with our ears

We listen with our brain trough our ears. Our brain interpret what the ears are sending, but not always the same way, depending on many aspect, one of them being what are concentrating on. Hard to trust what our own brain is interpreting when more than the ears are engaged for various reason that have been demonstrated scientifically.

Ultimately, audibility and perception are the only things that matter, surely?

Surely not, particularly in trying to share objective information about anything related to earring.
Are they enough to enjoy listening to music? Absolutely, as long as it relates only to one self in a particular set of circumstances. No one here will ever tell you what to enjoy, you are on your own for that.
Claiming to ear differences where measurements says their is no audible differences, that will require proof of concept, in your case that when your brain is engage, the only factor influencing your brain interpretation are your ears. Sounds fair enough to me.

 

[manisandher]

If PKMetric increases the error, this would be a very strange case where the device produces an error in the high audibility frequency range while decreasing it in the range where it’s normally not audible. Hard to imagine such a device, and can only happen if you use dBr units, since that’s relative to the signal level. dBFS PKMetric should not exceed RMS of the null signal.

The Prism ADC was displaying exactly this behaviour with some results. Might having its noise-shaping switched off have caused this?

 

[manisandher]

If that’s the only thing I get to go by, then I’d pick the -70dB PKMetric.

Thanks.

But a single number is never going to tell the whole story, so I’d prefer to look at the full PKMetric plot over the entire time range.

Is there a way of seeing how the PK Metric changes with frequency?

 

[manisandher]

Our brain interpret what the ears are sending, but not always the same way, depending on many aspect, one of them being what are concentrating on.

Agree. And therefore why ABX test results need to be taken with a pinch of salt. If you score well consistently, it's very likely that you're hearing a difference. If you don't, you may or may not be capable of hearing a difference - it may just be that your brain isn't well-enough engaged, for whatever reason.

FWIW, I found the blind listening test I was involved with a few years back absolutely fatiguing, and nothing like the enjoyable experience of listening to good-sounding music.

 

[pkane]

Thanks.



Is there a way of seeing how the PK Metric changes with frequency?

PK Metric is a time-based metric, doing 400ms SFFTs. For each 400ms interval, you can see the frequency content if you open the double-side view from the icon near top right. To select different time intervals, hold down the Ctrl key while moving the mouse over the desired time you want to review. The frequency content of the section under the mouse pointer will be displayed in the right window.

 

[manisandher]

PK Metric is a time-based metric, doing 400ms SFFTs. For each 400ms interval, you can see the frequency content if you open the double-side view from the icon near top right. To select different time intervals, hold down the Ctrl key while moving the mouse over the desired time you want to review. The frequency content of the section under the mouse pointer will be displayed in the right window.

Oh, that's great. Will try this when I get back.

 

[amirm]

I suspect many people visiting this forum get the impression that SINAD is the only thing they need to be concerned about. Indeed, how many times has our host here (who I have a lot of respect for) and others claimed that once you've reached a SINAD of 115dB, everything is sorted as far as audibility is concerned?

SINADS at 115 dB and higher are strictly noise dominated. Distortion from such devices routinely clocks at 120 to -135 dB. Since the noise is measured across the full audible spectrum, this indicates inaudibility of noise regardless of test signal. On distortion, for the specific test where the comment is made, i.e. 1 khz, that is also full transparency. Such DACs also do superbly in THD+N tests relative frequency. And at any rate, listeners are horrible at detecting non-linear distortions in electronics. All of this points to high confidence statement of transparency base on what we know of human hearing.

Back to your OP, none of this matters unless you, as the only person hearing such a difference, demonstrates the same using ABX test. Until that is confirmed, it makes no sense to chase the "why" when the "what" is not established.

 

[gvl]

This whole discussion started on the premise that the dCS sounds obviously different than the other two DACs. There is not an indication of this happening in the samples provided. There may be a difference my ears not detecting but clearly it's not as big as the OP may suggest. There was a similar discussion that went over 70 pages, many man-hours lost due to the OP neglecting to do a blind test, and in the end there was no clear answer anyway. Do a blind test, accept the reality, and move on.

Both DAC are transparent as measures by Amir, why one not as clean sounding as the other?

All, Before you start typing words like "brain", "imagination", please save your energy. I won't response to such replies. I have done multiple online blind tests, done multiple ABX tests, and a coupe of DAC vs DAC double blind tests. So, please save the lectures about voltage match...

www.audiosciencereview.com

 

[manisandher]

Do a blind test, accept the reality, and move on.

I'm not going to do one, for the reasons I've cited a number of times now.

If this thread is therefore of no interest to you, just move along.