• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

A More Useful Way of Measuring the Performance of DACs?

Status
Not open for further replies.

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
I recently acquired a 13-year-old dCS Scarlatti CD transport, DAC, clock and upsampler.

What seemed immediately apparent to me was how different it sounded to the two modern DACs I have here: an Okto dac8PRO and an RME ADI-2 Pro BE. I think I hear very subtle differences between these modern DACs – the RME sounds really ‘dynamic’, and the Okto very ‘clean’… but I’m not confident that I’d be able to pass an ABX test with one against the other. However, against the dCS, the differences are obvious – the dCS sounds way more laid back, with a fuller bottom end. It sounds nowhere near as incisive as the modern DACs, almost to the point of sounding boring in comparison. I’m 100% certain I’d pass an ABX with the dCS against either of these modern DACs.

In any event, I wanted to explore what might be contributing to the differences I’m hearing. So, I set up the following 2 chains:
  1. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> RME ADI-2 Pro BE @+13dBu -> USB -> RME Digicheck (on PC)
  2. Roon (64-bit level-matching) -> RJ45 -> Auralic Aries G1.1 @0dB -> USB -> DUT @0dB -> XLR analogue output (2V) -> Prism AD124 @+12dBu -> BNC -> Tascam DA3000 (SD card)
The DUT and ADC clocks were synchronised in all cases.

I played three different 24/44.1 tracks and captured the analogue outputs at 24/44.1 in order to perform null tests with @pkane 's superb DeltaWave software. The only settings engaged in DeltaWave were:
  • match gain: ON
  • subsample offset: ON
All other settings were OFF.

1. 1kHz sine @0dBFS

From measurements posted on ASR, the Okto has a SINAD of around 118dB, and the RME around 115dB, which place them both in ASR’s ‘excellent’ category.

Using the RME as ADC, I get the following results:

1. Okto 1kHz.jpg
2. RME 1kHz.jpg

Both of these have higher 3rd harmonics than the ASR measurements, which is probably down to the ADC, but are otherwise in-line with expectations, I think.

And here’s the FFT for the dCS:

3. dCS 1kHz.jpg

Clearly, the modern DACs measure better. So, could this account for the differences that I hear between them and the dCS? I suspect most people here would say “no”, as the differences seem too low in level to be audible.

Could DeltaWave provide any insights? Well, here are the results:

DeltaWave - 1kHz sine 0dB.JPG

There are two surprises, to my mind:
  1. The dCS measures very closely to the modern DACs. Perhaps the ADCs are the limiting factors here?
  2. The 25-year-old Prism is bloody brilliant for its age, achieving the best null at -100.46dBA!

2. Gearspace ‘Original2’

This is the track used on the Gearspace loopback thread: https://gearspace.com/board/gear-sh...ing-ad-da-loops-means-audio-diffmaker-76.html. This is real music, and not a test signal.

Here are the DeltaWave results:

DeltaWave - Original2.JPG

Two more surprises here:
  1. The RMS differences put the DAC/ADC combos here level with best-of-the-best in the Gearspace test!
  2. Again, the Prism does really well, achieving better RMS nulls than the RME!

Especially with the old-timer ADC, the modern DACs are clearly achieving better RMS nulls than the dCS, so they're better, right? Well, it’s worth looking at the dBA and PK Metric scores in more detail.

dBA is a dB measurement that has been adjusted to take into account the varying sensitivity of the human ear to different frequencies of sound. Low and very high frequencies are given less weight than on the standard dB scale:
dBA-weighting-logarithmic.png
(from http://wordpress.mrreid.org/2014/02/27/dba-and-grey-noise/)

With the Prism, the dCS now measures the best in dBA.

The PK Metric is a (very clever) metric that @pkane has created that represents a much more perceptually-weighted result than the RMS Difference – it should be more audibly and perceptually accurate. You can read more about it here: https://deltaw.org/pk_metric.html.

With the Prism, but especially with the RME as ADC, the dCS now pulls aways substantially on the PK Metric score. Audibly and perceptually, it seems that the dCS is more accurate than the modern DACs!

I took many multiple captures with all the DACs, and these results are totally repeatable and consistent. But perhaps the dCS only does well with this particular classical music track? So, I repeated the procedure with another track…

3. Diana Krall – Temptation

Here are the DeltaWave results:

DeltaWave - Temptation.JPG

No... it's definitely the DAC and not the track!

******************************

I'd like to do more testing, but these initial results have surprised me.

Any thoughts/insights welcome.

Cheers,
Mani.
 

gvl

Major Contributor
Joined
Mar 16, 2018
Messages
3,211
Likes
3,556
Location
SoCal
I’m 100% certain I’d pass an ABX with the dCS against either of these modern DACs.

It does seem to be a bit convoluted to go through all the trouble collecting data without actually trying a blind test first, sounds like you have track captures already, just use the Foobar ABX tool or the like and see if the certainty you think there is means anything.
 

DVDdoug

Major Contributor
Joined
May 27, 2021
Messages
1,513
Likes
1,929
but I’m not confident that I’d be able to pass an ABX test with one against the other. However, against the dCS, the differences are obvious
If it's obvious you can ABX it every time! ;)

– the dCS sounds way more laid back
Nobody knows what that means...

with a fuller bottom end
A difference in bass should be easy to measure.

An audible frequency difference would be unusual.

The most common difference* would be noise. (BTW - digital "glitches" usually sound like noise, but that's usually something in the transmission/connection rather than the DAC itself.)

The only other difference would be distortion. Audible distortion from a DAC is also unusual, especially with music or other normal program material.

RME sounds really ‘dynamic’,
DACs are linear so they don't really affect the dynamics.



* Actually the most likely difference is output level... Usually that's not what we are listening-for so ABX tests are normally level-matched. But higher output (louder) is often heard as "more dynamic"..
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
It does seem to be a bit convoluted to go through all the trouble collecting data without actually trying a blind test first, sounds like you have track captures already, just use the Foobar ABX tool or the like and see if the certainty you think there is means anything.
If it's obvious you can ABX it every time! ;)

I've done my 'ABX time'. Scored 9/10 in a blind test... and still people didn't believe I heard what I did. So, no thanks.

It's interesting that both of you have picked up on my subjective listening comments, but not on the objective measurements I've taken. A 13-year-old DAC measures more audibly and perceptually accurate than a couple of well-measuring modern DACs. Doesn't that intrigue you in any way?
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
* Actually the most likely difference is output level... Usually that's not what we are listening-for so ABX tests are normally level-matched. But higher output (louder) is often heard as "more dynamic"..

No.

All 3 DACs level-matched to 0.01dB using Roon's DSP.
 

sonitus mirus

Active Member
Joined
Apr 19, 2021
Messages
148
Likes
195
Interesting stuff, thanks for sharing.

Which filters were used on the DACs? Could the differences be mostly above 12KHz with the music? It could be the way the filters are working, rolling off or sharply cutting off.
 
  • Like
Reactions: gvl

gvl

Major Contributor
Joined
Mar 16, 2018
Messages
3,211
Likes
3,556
Location
SoCal
I've done my 'ABX time'. Scored 9/10 in a blind test... and still people didn't believe I heard what I did. So, no thanks.

It's interesting that both of you have picked up on my subjective listening comments, but not on the objective measurements I've taken. A 13-year-old DAC measures more audibly and perceptually accurate than a couple of well-measuring modern DACs. Doesn't that intrigue you in any way?

The results piqued my interest but it's possible there is a fault with your metrology. It would be interesting to dig deeper into the differences and try to understand the contributing factors. But in the end, if you can't ABX them reliably, does it really matter? The RMS tests are very close. It's just statements like "I'm 100% sure I'd be able to recognize the differences" is a big red flag here that steers discussions in a certain way.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Have you measured magnitude and phase response? Seems most likely differences can be attributed to magnitude / phase response as a result of different filters.

Michael

Here are the phase responses of the DACs into the Prism, with the Krall track:

Delta Phase - RME.jpgDelta Phase - Okto.jpgDelta Phase - dCS.jpg

Note the different scales.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
Which filters were used on the DACs?

I used the steepest/fastest filters available in each of the DACs. Here's the FR response for the Krall track:

FR - DAC A.JPGFR - DAC B.JPGFR - DAC C.JPG
 

mdsimon2

Major Contributor
Joined
Oct 20, 2020
Messages
1,767
Likes
2,188
Location
Detroit, MI

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
4,831
Likes
7,822
Location
Europe
All 3 DACs level-matched to 0.01dB using Roon's DSP.
During the ABX test or for sighted listening? Beware you need to do both, levels matched and blind, not only one.

Regarding your impressions: a fuller bass should be seen easily in a frequency sweep, or in high harmonic distortion in a 20 Hz or 50 Hz signal test.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
How about a frequency response sweep in REW?

It's been many years since I last used REW. I'll see if I can download and get my head around it quickly.

Mani.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
During the ABX test or for sighted listening? Beware you need to do both, levels matched and blind, not only one.

Yes, of course. But I have no interest in ABX testing for the reasons cited earlier.
 

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
4,831
Likes
7,822
Location
Europe
I've done my 'ABX time'. Scored 9/10 in a blind test... and still people didn't believe I heard what I did. So, no thanks.
Sorry to insist. If differences are sooo obvious I'd expect a 100% success rate. With 9/10 you failed once to notice an obvious difference hence I think it's not that obvious in reality as you think.

I did such a blind test between 2 DACs and scored 8/10, and although the sighted test was very obvious the blind test showed me how much expectation bias influenced my experience. The difference in the blind test was so miniscule I knew immediately I could live with both DACs.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK
What people seem to be missing is the PK Metric really is the gold standard for audibility and perception.
 

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
4,831
Likes
7,822
Location
Europe
Yes, of course. But I have no interest in ABX testing for the reasons cited earlier.
Well, skipping it may end up fooling yourself, because - sadly - we humans are best fooling ourself.
 

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
4,831
Likes
7,822
Location
Europe
What people seem to be missing is the PK Metric really is the gold standard for audibility and perception.
But it does not explain fuller sound. It might explain the laid back character due to less audible distortion.
 
OP
manisandher

manisandher

Addicted to Fun and Learning
Joined
Nov 6, 2016
Messages
523
Likes
429
Location
Royal Leamington Spa, UK

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
4,831
Likes
7,822
Location
Europe
It's been many years since I last used REW. I'll see if I can download and get my head around it quickly.

Mani.
If you succeed try a multitone test. If you miss the test signal I can provide one. Just let me know samplerate and length.
 
Status
Not open for further replies.
Top Bottom