A More Useful Way of Measuring the Performance of DACs?

[manisandher]

But it does not explain fuller sound. It might explain the laid back character due to less audible distortion.

If what I'm hearing really exists, then it must be measurable, and the PK Metric is the best bet.

 

[sonitus mirus]

If what I'm hearing really exists, then it must be measurable, and the PK Metric is the best bet.

You don't have to share your results with us, but did you try and ABX test to verify that you are hearing differences? You keep saying it is obvious, but you have not stated that you actually conducted a personal test to validate your assessment.

 

[manisandher]

You don't have to share your results with us, but did you try and ABX test to verify that you are hearing differences? You keep saying it is obvious, but you have not stated that you actually conducted a personal test to validate your assessment.

As I've stated before... I went to a lot of trouble to do blind ABX a few years ago. It took 18 minutes to conduct. Scored 9/10 (got #9 wrong). Still wasn't believed that I was hearing what I was. So... never again.

 

[sonitus mirus]

As I've stated before... I went to a lot of trouble to do blind ABX a few years ago. It took 18 minutes to conduct. Scored 9/10 (got #9 wrong). Still wasn't believed that I was hearing what I was. So... never again.

But, we don't even know if you are actually hearing any differences at all. A test is a test and you identify differences or you don't. That is not up for debate. We may contest the reason for differences being heard, but whether you are hearing them or not is not the issue.

 

[manisandher]

Humour me. Take a listen to the 3 files I linked. One should sound 'duller' than the other two. That's the dCS. The other two sound similar.

That's what I'm hearing with the captures. It's more pronounced listening directly to the outputs of the DACs (level-matched, blah, blah, blah).

 

[dominikz]

@manisandher I'm sure you don't want to read another one of this kind of post, but it is written in good faith and I hope you could still find it useful and won't mind.

I've done my 'ABX time'. Scored 9/10 in a blind test... and still people didn't believe I heard what I did. So, no thanks.

You can't really blame people for being skeptical about these kind of claims - there's overwhelming evidence people in general can't reliably identify differences between reasonably well-measuring audio electronics. A common approach in this case is to follow the good old maxim "extraordinary claims require extraordinary evidence".
Otherwise people could be spending time looking for causes of differences that might disappear in a rigorously controlled listening test.
It is simply good research practice to first establish the observed effect is real without any shadow of a doubt (and repeatable), before diving in and looking for its cause.

Also, scoring 9/10 one time in a controlled ABX test is not that remarkable, nor proof of being able to reliably hear a difference - e.g. flipping a coin enough times is bound to generate a sequence of 9 out of 10 tails in a row at some point, and maybe even on the first 10 if you're lucky.

On the other hand, scoring 9/10 (or 10/10) many times in a row would be a strong indicator there was an audible difference you could hear; but again it couldn't exclude issues with setup or methodology (which are unfortunately common because setting up rigorous tests is difficult).

What people seem to be missing is the PK Metric really is the gold standard for audibility and perception.

I'm sorry to ask - but do you have references to research formally linking PK Metric results to results of perceptual studies / listening tests?

Don't get me wrong - I'm a fan of the tool (and in general the amazing work and resources provided by @pkane), but I'm not convinced the figures you show in your first post really have predictable correlation with audibility. However I'm happy to be proven wrong and to learn something new!

E.g. in my online ABX test of two very different DACs the PK Metric gave -56dBrms for the two files, yet the results of over 350 participants do not indicate differences were audibly significant for the overwhelming majority of participants. And in this case the differences between DACs were ~30dB of SINAD and one even had a significant frequency response deviation - so quite significant (I was actually expecting a very different distribution)!

Humour me. Take a listen to the 3 files I linked. One should sound 'duller' than the other two. That's the dCS. The other two sound similar.

Sorry I might be blind - but I can't see the links to the source audio files recorded with the 3 DACs in any of the posts

EDIT: Sorry, missed the post - found it now!

 

[sq225917]

Be my guest:

DAC A: https://drive.google.com/file/d/1pDqYOz55NOWQrc8X5iN52SwPrZy2pI0X/view?usp=share_link
DAC B: https://drive.google.com/file/d/1iI4BD70b1oAJ049sU13ytziBbwxPqH1p/view?usp=share_link
DAC C: https://drive.google.com/file/d/1-SdDrytUsUojfkKE5LZbMEMR6nOtwL0Z/view?usp=share_link

Here

 

[manisandher]

@manisandher I'm sure you don't want to read another one of this kind of post, but it is written in good faith and I hope you could still find it useful and won't mind.

Yes, sure.

Also, scoring 9/10 one time in a controlled ABX test is not that remarkable...

There's a 1/100 chance of achieving a 9/10 in an ABX.

I claimed I could hear a difference (between the same file played back in two different, but totally bit-identical ways) and was told it was impossible. So, I invited someone up to help me conduct a blind ABX. We agreed that the results would be posted online, irrespective of the outcome. I scored 9/10 on my first and only ABX. Phew. The Gods must have been smiling down on me that day.

[In full disclosure, the ABX test was actually the 3rd of 3 blind listening tests we conducted that day. The first two were non-ABX though. Bear in mind that I had never been involved in any blind listening tests prior to this, and it took a while to get accustomed. In any event, here are the full results for you to make of what you will:

]

Mani.

 

[sonitus mirus]

Humour me. Take a listen to the 3 files I linked. One should sound 'duller' than the other two. That's the dCS. The other two sound similar.

That's what I'm hearing with the captures. It's more pronounced listening directly to the outputs of the DACs (level-matched, blah, blah, blah).

I can't hear any obvious differences, certainly nothing that I would be concerned about unless there is a specific spot in the track that is a problem that I am not aware about, as I did not listen to the full track, just short sections of it in a few places.

 

[manisandher]

Be my guest:

DAC A: https://drive.google.com/file/d/1pDqYOz55NOWQrc8X5iN52SwPrZy2pI0X/view?usp=share_link
DAC B: https://drive.google.com/file/d/1iI4BD70b1oAJ049sU13ytziBbwxPqH1p/view?usp=share_link
DAC C: https://drive.google.com/file/d/1-SdDrytUsUojfkKE5LZbMEMR6nOtwL0Z/view?usp=share_link

Listen to DK's voice and to the electric guitar throughout the track. The Okto gives the impression of the most reverb, and the dCS of the least reverb. Yes, it's subtle in these captures. More obvious listening directly to the DACs.

Mani.

 

[dominikz]

There's a 1/100 chance of achieving a 9/10 in an ABX.

Sure, but even unlikely events happen by chance. Actually, they happen all the time. E.g. notice that normal distribution describes random/chance behavior and it contains events as rare (or rarer) than those happening in 1% of cases. Notice that I'm not saying yours is a chance results, I'm just saying there's not enough evidence to be certain it isn't.

Phew. The Gods must have been smiling down on me that day.

I'd reckon random processes are quite unaware that some sequences are considered special by humans Though it is not 100% apples-to-apples, it reminds me of a R. Feynman quote:

You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won't believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!

I joke, but in principle no serious research will end after a single successful trial with a 9/10 result. It is simply not enough to be called 'proof'. In addition, and as I said previously, there are always very real dangers of issues in methodology or test setup that can skew results in the first place.

Perhaps you will enjoy this post:

I just flipped a coin 16 times.
6x heads, 10x tails

Also threw dice 16x and counted odd/even and did this 4 times
1: 7 even, 9 odd
2: 9 even, 7 odd
3: 10 even, 6 odd
4: 12 even, 4 odd

Notice that the probability to get 12 even out of 16 dice throws by chance is 4/100 - also not very likely, but it happens.

In conclusion, you may hear a difference and don't need more proof than that to decide it is real - that is of course absolutely fine. Just don't be surprised if convincing other people your results and conclusions are valid takes more effort than that.

BTW I quickly listened to the 3 files and heard no obvious difference. I also didn't see anything I'd consider audibly significant when loading the files in REW or my DAW - after this I have to admit I lost interest.

Anyway, since I have nothing else to contribute I'll disengage. Good luck!

 

[manisandher]

I can't hear any obvious differences, certainly nothing that I would be concerned about unless there is a specific spot in the track that is a problem that I am not aware about, as I did not listen to the full track, just short sections of it in a few places.

Thanks for trying.

No, there's nothing to be concerned about with any of the DACs. I'm very happy with the RME and Okto and would recommend them heartily to anyone looking for a good DAC. I'm still getting used to the sound of the dCS.

 

[manisandher]

Notice that the probability to get 12 even out of 16 dice throws by chance is 4/100 - also not very likely, but it happens.

4/100 isn't 1/100 .

BTW I quickly listened to the 3 files and heard no obvious difference. I also didn't see anything I'd consider audibly significant when loading the files in REW or my DAW - after this I have to admit I lost interest.

Thanks for trying.

 

[tuga]

DACs are linear so they don't really affect the dynamics.

"Sounds really 'dynamic'" implies perceived dynamics.

 

[dominikz]

4/100 isn't 1/100 .

It sure isn't

 

[restorer-john]

All the people clamouring for ABX tests- do them yourselves and stop being lazy!

@manisandher , well done presenting your findings. I agree with you, having done way too many pointless ABX tests- they are no fun and prove nothing to anyone but yourself and even then, they really don't help with preferences. Line up 10 amplifiers and compare them in real time, level matched. ABX is about as useful as an ashtray on a motorbike in the real world.

 

[manisandher]

"Sounds really 'dynamic'" implies perceived dynamics.

Yes.

But here's something I find interesting...

When the RMS levels of the 3 captures are matched to within 0.001dB, their peak levels differ by 0.1-0.2dB. This must be down to the different dynamic behaviour of the 3 DACs... something that is not revealed in any of the tests currently conducted here on ASR, or elsewhere, that I can tell.

 

[tuga]

Yes.

But here's something I find interesting...

When the RMS levels of the 3 captures are matched to within 0.001dB, their peak levels differ by 0.1-0.2dB. This must be down to the different dynamic behaviour of the 3 DACs... something that is not revealed in any of the tests currently conducted here on ASR, or elsewhere, that I can tell.

It would be interesting to determine if there is in fact a correlation between the two. Maybe @j_j could help?

 

[sonitus mirus]

Yes.

But here's something I find interesting...

When the RMS levels of the 3 captures are matched to within 0.001dB, their peak levels differ by 0.1-0.2dB. This must be down to the different dynamic behaviour of the 3 DACs... something that is not revealed in any of the tests currently conducted here on ASR, or elsewhere, that I can tell.

It would be interesting to note if similar differences might exist between two exact models, or even the same device under test if measurements were taken multiple times.

 

[mdsimon2]

All the people clamouring for ABX tests- do them yourselves and stop being lazy!

@manisandher , well done presenting your findings. I agree with you, having done way too many pointless ABX tests- they are no fun and prove nothing to anyone but yourself and even then, they really don't help with preferences. Line up 10 amplifiers and compare them in real time, level matched. ABX is about as useful as an ashtray on a motorbike in the real world.

While I agree in sentiment (and have certainly dismissed equipment based on preference without blind ABX testing) it is a bit different to claim a sound preference AND correlation to the pkmetric with minimal controls. Not to mention the lack of exploration of basic tests such as magnitude and phase response before jumping to a conclusion that the pkmetric shows us something that basic frequency response measurements do not.

When the RMS levels of the 3 captures are matched to within 0.001dB, their peak levels differ by 0.1-0.2dB. This must be down to the different dynamic behaviour of the 3 DACs... something that is not revealed in any of the tests currently conducted here on ASR, or elsewhere, that I can tell.

Again this could be easily explained by differing frequency response.

Also the level matching methodology was not shared. For listening tests was it done at the amplifier output terminals? If not differences in output impedance of the DAC could have a very real impact on level. Also if you level match based on the ADC measurements it is quite possible you will not be level matched during listening tests due to differences in input impedance between the amplifier and ADC.

If I was the OP I think I would just enjoy my gear and be happy and not try to sway the opinion of folks who are unlikely to change their mind, YMMV.

Michael