• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

A More Useful Way of Measuring the Performance of DACs?

Status
Not open for further replies.
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Would you be testing the original file vs a recording or compare 2 recordings and consider one 'reference' ?

Against the original file.
 

danadam

Addicted to Fun and Learning
Joined
Jan 20, 2017
Messages
717
Likes
1,101
You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then analyse these with DeltaWave. You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -70dB
- PK Metric = -50dB

Which DAC would you be inclined to keep for your long-term listening pleasure?
The one time I happened to evaluate PK Metric, it hadn't worked that well for me. I didn't hear a difference at the point where it indicated -47 or -52 dBFS difference, but had no problem at one point with -56 dBFS difference.
 

Scytales

Member
Joined
Jan 17, 2020
Messages
41
Likes
54
Location
France
[...]

In any event, I wanted to explore what might be contributing to the differences I’m hearing. So, I set up the following 2 chains:
  1. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> RME ADI-2 Pro BE @+13dBu -> USB -> RME Digicheck (on PC)
  2. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> Prism AD124 @+12dBu -> BNC -> Tascam DA3000 (SD card)
[...]

I played three different 24/44.1 tracks and captured the analogue outputs at 24/44.1 in order to perform null tests with @pkane 's superb DeltaWave software.

1. 1kHz sine @0dBFS

[...]

Could DeltaWave provide any insights? Well, here are the results:

View attachment 245311

[...]

2. Gearspace ‘Original2’

This is the track used on the Gearspace loopback thread: https://gearspace.com/board/gear-sh...ing-ad-da-loops-means-audio-diffmaker-76.html. This is real music, and not a test signal.

Here are the DeltaWave results:

View attachment 245312

[...]

3. Diana Krall – Temptation

Here are the DeltaWave results:

View attachment 245316

Hi.

Thanks for your efforts.

However, I'm a bit lost by your first message.

I understand you made two different captures of the analogue output of each of your three DACs from the same test files with two different recording chains, each incorporating a different ADC. That's of no concern to me.

What bothered me is that I have not read anywhere which two captures have been compared with each other to obtain each and every null difference, whatever the metric or the recording chain.

On the Deltawave software website, it is explained that said software "allow detailed comparison of two different captures of the same audio signal".

OK, but what are the two different captures you have compared in Deltaware to produce the above cited charts ?
 

danadam

Addicted to Fun and Learning
Joined
Jan 20, 2017
Messages
717
Likes
1,101
OK, but what are the two different captures your have compared in Deltaware to produce the above cited charts ?
As I understand this, these are the captures compared to the original file.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Thanks for your effort.

Thanks.
What bothered me is that I have not read anywhere which two capture have been compared with each other to obtain each and every null difference, whatever the metric or the recording chain.

Each capture was compared to the original file that was fed into the DACs.

Mani
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
On reflection, the question I posed earlier is pretty dumb. It's very unlikely that the PK Metric would ever be worse than the RMS Difference, and certainly not substantially worse.
 

pkane

Major Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
4,487
Likes
7,752
Location
North-East
A quick question for anyone who's looked into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then analyse these with DeltaWave. You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -70dB
- PK Metric = -50dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

Mani.

PKMetric starts with the RMS difference. In most DACs, the large RMS error is due to filters (reconstruction, anti-imaging and DC). Because these filters change FR around extremes, PKMetric will lower the error, not increase it by applying frequency-audibility threshold weights.

If PKMetric increases the error, this would be a very strange case where the device produces an error in the high audibility frequency range while decreasing it in the range where it’s normally not audible. Hard to imagine such a device, and can only happen if you use dBr units, since that’s relative to the signal level. dBFS PKMetric should not exceed RMS of the null signal.

Edit: I see you posted the same :)
 

FINFET

Member
Joined
Jul 27, 2022
Messages
45
Likes
54
Using real music file diffs for dac measurements (instead of tone/multitones) could be very tricky. I once posted some similar question (but not with good measurements and experiments like op) and start to gradually understand the difficulty later about how filters and phase change could affect music file nulling. Some posts by members in that thread might interest you as well.
 

pkane

Major Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
4,487
Likes
7,752
Location
North-East
Archimago also started doing this a while ago (but not with Deltawave) but abandonned it.
There is also or Russian guy on this forum pushing his difference files. I believe just before Paul came with Deltawave. I am sure @pkane knows who this is. He already had quite a collection of data on phones, DACs, MP3 players.

When it is not an official standard way of testing I don't think Amir will adopt it.



The one with -50 is potentially audible different.
Whether it is RMS or PK metric it would warrant an investigation.
Would you be testing the original file vs a recording or compare 2 recordings and consider one 'reference' ?
Serge Smirnoff proposed a DF metric, which is similar to the RMS null and has no perception-based weights applied to the result. DeltaWave also computes DF Metric, so you have your choice of which metric to use :)
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Take Two...

For those of you who've had a chance to look into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then compare the captures vs. the original in DeltaWave (using only 'match gain' and 'subsample offset' settings). You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -60dB
- PK Metric = -60dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

@pkane , I'd love to hear which you'd choose, and why.

Mani.
 

pkane

Major Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
4,487
Likes
7,752
Location
North-East
Take Two...

For those of you who've had a chance to look into the PK Metric...

You are given two DACs to home test by a dealer. You take some captures with a good-quality ADC, and then compare the captures vs. the original in DeltaWave (using only 'match gain' and 'subsample offset' settings). You get the following (totally repeatable and consistent) results across a whole range of different musical genres:

DAC A:
- RMS Difference = -50dB
- PK Metric = -70dB
DAC B:
- RMS Difference = -60dB
- PK Metric = -60dB

Which DAC would you be inclined to keep for your long-term listening pleasure?

@pkane , I'd love to hear which you'd choose, and why.

Mani.

If that’s the only thing I get to go by, then I’d pick the -70dB PKMetric. But a single number is never going to tell the whole story, so I’d prefer to look at the full PKMetric plot over the entire time range.
 

Cote Dazur

Senior Member
Joined
Feb 25, 2022
Messages
414
Likes
376
Location
Canada
We listen to music with our ears
We listen with our brain trough our ears. Our brain interpret what the ears are sending, but not always the same way, depending on many aspect, one of them being what are concentrating on. Hard to trust what our own brain is interpreting when more than the ears are engaged for various reason that have been demonstrated scientifically.
Ultimately, audibility and perception are the only things that matter, surely?
Surely not, particularly in trying to share objective information about anything related to earring.
Are they enough to enjoy listening to music? Absolutely, as long as it relates only to one self in a particular set of circumstances. No one here will ever tell you what to enjoy, you are on your own for that.
Claiming to ear differences where measurements says their is no audible differences, that will require proof of concept, in your case that when your brain is engage, the only factor influencing your brain interpretation are your ears. Sounds fair enough to me. :)
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
If PKMetric increases the error, this would be a very strange case where the device produces an error in the high audibility frequency range while decreasing it in the range where it’s normally not audible. Hard to imagine such a device, and can only happen if you use dBr units, since that’s relative to the signal level. dBFS PKMetric should not exceed RMS of the null signal.

The Prism ADC was displaying exactly this behaviour with some results. Might having its noise-shaping switched off have caused this?
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
If that’s the only thing I get to go by, then I’d pick the -70dB PKMetric.

Thanks.

But a single number is never going to tell the whole story, so I’d prefer to look at the full PKMetric plot over the entire time range.

Is there a way of seeing how the PK Metric changes with frequency?
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Our brain interpret what the ears are sending, but not always the same way, depending on many aspect, one of them being what are concentrating on.

Agree. And therefore why ABX test results need to be taken with a pinch of salt. If you score well consistently, it's very likely that you're hearing a difference. If you don't, you may or may not be capable of hearing a difference - it may just be that your brain isn't well-enough engaged, for whatever reason.

FWIW, I found the blind listening test I was involved with a few years back absolutely fatiguing, and nothing like the enjoyable experience of listening to good-sounding music.
 

pkane

Major Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
4,487
Likes
7,752
Location
North-East
Thanks.



Is there a way of seeing how the PK Metric changes with frequency?
PK Metric is a time-based metric, doing 400ms SFFTs. For each 400ms interval, you can see the frequency content if you open the double-side view from the icon near top right. To select different time intervals, hold down the Ctrl key while moving the mouse over the desired time you want to review. The frequency content of the section under the mouse pointer will be displayed in the right window.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
PK Metric is a time-based metric, doing 400ms SFFTs. For each 400ms interval, you can see the frequency content if you open the double-side view from the icon near top right. To select different time intervals, hold down the Ctrl key while moving the mouse over the desired time you want to review. The frequency content of the section under the mouse pointer will be displayed in the right window.

Oh, that's great. Will try this when I get back.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
39,960
Likes
185,366
Location
Seattle Area
I suspect many people visiting this forum get the impression that SINAD is the only thing they need to be concerned about. Indeed, how many times has our host here (who I have a lot of respect for) and others claimed that once you've reached a SINAD of 115dB, everything is sorted as far as audibility is concerned?
SINADS at 115 dB and higher are strictly noise dominated. Distortion from such devices routinely clocks at 120 to -135 dB. Since the noise is measured across the full audible spectrum, this indicates inaudibility of noise regardless of test signal. On distortion, for the specific test where the comment is made, i.e. 1 khz, that is also full transparency. Such DACs also do superbly in THD+N tests relative frequency. And at any rate, listeners are horrible at detecting non-linear distortions in electronics. All of this points to high confidence statement of transparency base on what we know of human hearing.

Back to your OP, none of this matters unless you, as the only person hearing such a difference, demonstrates the same using ABX test. Until that is confirmed, it makes no sense to chase the "why" when the "what" is not established.
 

gvl

Major Contributor
Joined
Mar 16, 2018
Messages
3,211
Likes
3,556
Location
SoCal
This whole discussion started on the premise that the dCS sounds obviously different than the other two DACs. There is not an indication of this happening in the samples provided. There may be a difference my ears not detecting but clearly it's not as big as the OP may suggest. There was a similar discussion that went over 70 pages, many man-hours lost due to the OP neglecting to do a blind test, and in the end there was no clear answer anyway. Do a blind test, accept the reality, and move on.

 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Do a blind test, accept the reality, and move on.

I'm not going to do one, for the reasons I've cited a number of times now.

If this thread is therefore of no interest to you, just move along.
 
Status
Not open for further replies.
Top Bottom