A More Useful Way of Measuring the Performance of DACs?

[manisandher]

Based on the plots in #10.

The plots don't identify the DACs. You've just made an assumption that the dCS is DAC C.

 

[Sokel]

Because the matter of dbr vs dbrA is a matter of a little controversy here:
In a typical 1Khz SINAD measurement what would be the difference numerically?

 

[voodooless]

The plots don't identify the DACs. You've just made an assumption that the dCS is DAC C.

Ah, wait, thought they were annotated, my bad The argument may still hold if the other DAC ABX'ed was the one with the earlier cut-off.

 

[manisandher]

What were the gains that you entered in Roon for the three DAC's?

RME: -4.58dB
Okto: -6.62dB
dCS: 0dB

I did this for thoroughness, but to be honest, having the DACs at different output levels had no effect on the null results (provided the ADC didn't go into clipping, of course).

 

[voodooless]

From the link:

"PK Metric is computed in the frequency domain over multiple, overlapping time windows of 400ms each. For each 400ms time window an STFT (Short-Time Fourier Transform) is performed for both, the reference and the comparison. The spectra of the two windows is corrected using equal loudness curves. This step uses an interpolated version of ISO 226:2003 curves and is designed to adjust the frequency weights according to their audibility at the current playback level. The level is estimated from the sound energy computed for each of the two windows being compared. For example, a 20kHz frequency will be weighted a whole lot less than 3kHz. The adjusted frequency responses are then reduced using an ERB (Equivalent Rectangular Bandwidth) smoothing filter, and finally, each of the ERB buckets in the comparison is subtracted from the same ERB bucket in the reference. The energy of the resulting error spectrum is then summed to produce a single dB value representing how loud the error is at that particular 400ms interval."

That's just a method of operation. None of this proves this is any gold standard..

Where's the 'circular reasoning'?

You declare it a gold standard -> Refer to the PK page -> PK tells us we should tell him if it's useful -> You declare it a gold standard

It doesn't get any more circular than that.

 

[j_j]

I think it might be a little FR difference making them audible.

Here is file A vs B. 1st linear and then log graphing. A is the reference file.

I doubt if this is audible on response, but that ultrasonic peak is well over 10 db. I am not clear on which DAC and ADC pairings were on the three files available. Possible aliasing or imaging is making a difference, but with such deep nulls it seems unlikely.
View attachment 245483

View attachment 245484

Now A vs C. I have an ADC that does this kind of rippled response. It seems borderline at best that it would be audible. But we have a ripple even down to below 3 khz and a slight droop above 10 khz. I've been able to discern this with Foobar ABX in the past (I haven't tried it on these files). Also this isn't the kind of thing where I listen to a couple songs and hear differences. I find some portions where I can hear a difference in 5 seconds of music with rapid switching. If I use 30 second segments I can hear no difference beyond chance.

View attachment 245487

View attachment 245486

Now B vs. C. The ripple is probably in C from what we've seen earlier. That might make A and B audible vs C. It also appears the ultrasonic difference is in B.

View attachment 245490

View attachment 245491

So I don't know if these ripples and ultrasonic issues make an audible difference. Here is A vs C with Frequency and Phase EQ engaged in Deltawave. I thought I remember being able to export the corrected audio file, but don't see it right now. Maybe @pkane could show us if I've simply forgotten how, and maybe @manisandher could correct all files for phase and FR then see if they still sound different.
View attachment 245494

View attachment 245493

View attachment 245495

Interesting, the first equiripple filter also has a small rolloff from something not-equiripple. The second filter appears to be one of a windowed sinc FIR, an IIR of some sort, or some unspecified FIR design that's not equiripple. Not being an equiripple filter puzzles me a touch if it's an FIR. Even an apodized FIR is equiripple The quaintness in the first filter is that the last two ripples are dropping off a bit, like there's a low-order IIR at some higher frequency. The quaintness of the second filter is in the lack of equiripple structure.

I'd flunk both, the second one more so.

 

[voodooless]

RME: -4.58dB
Okto: -6.62dB
dCS: 0dB

I did this for thoroughness, but to be honest, having the DACs at different output levels had no effect on the null results (provided the ADC didn't go into clipping, of course).

You did just lower the RME and Okto's dynamic range though. Again, I doubt any of this is audible but may influence your measurements.

 

[manisandher]

Because the matter of dbr vs dbrA is a matter of a little controversy here:

This is a key issue for me.

I suspect many people visiting this forum get the impression that SINAD is the only thing they need to be concerned about. Indeed, how many times has our host here (who I have a lot of respect for) and others claimed that once you've reached a SINAD of 115dB, everything is sorted as far as audibility is concerned? SINAD is calculated using a steady sine tone, not with a dynamically changing signal such as music. And I don't see multitone tests as being any better - great for measuring IMD, but not for audibility. I mean, have you ever tried listening to a multitone for more than a few seconds?

The current pool of measurements and tests conducted here are all necessary and absolutely valuable. But they do not give you the whole picture with regards to audibility and perception, IMHO. This is 'Audio Science Review', and should concern itself with all elements of audibility and perception, again IMHO.

 

[manisandher]

I doubt any of this is audible but may influence your measurements.

No, the null results don't change at all by disengaging Roon's DSP... provided ADC doesn't clip.

 

[solderdude]

Perception is a totally different field of expertise than measuring electronics and acoustic things though.
Maybe a new website is needed APS (Audio Perception Science) but the abreviation is already taken.

 

[manisandher]

Interesting, the first equiripple filter also has a small rolloff from something not-equiripple. The second filter appears to be one of a windowed sinc FIR, an IIR of some sort, or some unspecified FIR design that's not equiripple. Not being an equiripple filter puzzles me a touch if it's an FIR. Even an apodized FIR is equiripple The quaintness in the first filter is that the last two ripples are dropping off a bit, like there's a low-order IIR at some higher frequency. The quaintness of the second filter is in the lack of equiripple structure.

I'd flunk both, the second one more so.

See https://www.audiosciencereview.com/...ng-the-performance-of-dacs.39379/post-1388389.

 

[manisandher]

Perception is a totally different field of expertise than measuring electronics and acoustic things though.

Electronics, yes. But acoustics? Hmm.

We listen to music with our ears. Ultimately, audibility and perception are the only things that matter, surely? If not, this site should be called Audio Electronics Review, where SINAD is King.

 

[Sokel]

This is a key issue for me.

I suspect many people visiting this forum get the impression that SINAD is the only thing they need to be concerned about. Indeed, how many times has our host here (who I have a lot of respect for) and others claimed that once you've reached a SINAD of 115dB, everything is sorted as far as audibility is concerned? SINAD is calculated using a steady sine tone, not with a dynamically changing signal such as music. And I don't see multitone tests as being any better - great for measuring IMD, but not for audibility. I mean, have you ever tried listening to a multitone for more than a few seconds?

The current pool of measurements and tests conducted here are all necessary and absolutely valuable. But they do not give you the whole picture with regards to audibility and perception, IMHO. This is 'Audio Science Review', and should concern itself with all elements of audibility and perception, again IMHO.

Personally I run multitone vs level or freq vs SINAD on everything that falls in my hands,and even thought the level and structure of noise and distortion seems the same as Amir's measurements (with a lot of salt considering my amateur crap gear) the results in dbr scale are some 4-5db worst.
In a particular dac,with 1Khz SINAD over 104dbr (109dbrA in Amir's test) lows and highs are in the 100-ish for lows and in the 95-ish for very highs,so the dbA scale there would be most forgiving I think.

 

[voodooless]

And I don't see multitone tests as being any better - great for measuring IMD, but not for audibility. I mean, have you ever tried listening to a multitone for more than a few seconds?

Why does it matter that you don't like how multitone sounds? How is that a valid argument?

This is 'Audio Science Review', and should concern itself with all elements of audibility and perception, again IMHO.

Then let's focus on that, and do a few more ABX tests. That would go a long way toward convincing people that there is actually something to find out.

 

[solderdude]

Electronics, yes. But acoustics? Hmm.

We listen to music with our ears. Ultimately, audibility and perception are the only things that matter, surely? If not, this site should be called Audio Electronics Review, where SINAD is King.

Ears+Brain 'measure' different things than a Klippel, anechoic room, and even headphone measurements don't particularly correlate well with what's 'heard' even though some believe it always does or is more 'correct' than what specific measurements show.

Perception is not only the signal arriving but also includes all other senses as well as surrounding conditions. At least that's how I see this.

B.t.w. I applaud your efforts. Am agnostic about your findings and those of others.

And yes, I agree that the music quality/enjoyment of a person in the end is the only thing that matters.
Measurements are merely indicators of technical performance.

 

[manisandher]

You declare it a gold standard -> Refer to the PK page -> PK tells us we should tell him if it's useful -> You declare it a gold standard

I believe I hear differences between the dCS and the other DACs. SINAD measurements provide no insights. RMS null differences with 1kHz sine tone are similar for all DACs.

dBA null with real music alludes to perhaps there being differences between DACs. PK Metric with real music suggests clear differences, therefore the gold standard.

 

[manisandher]

Why does it matter that you don't like how multitone sounds? How is that a valid argument?

Huh? Because the only thing that matters to me is how music sounds. If a multitone test doesn't equate to this, then it's irrelevant to me... as a music listener (but certainly not as an electronics engineer, if I were one).

 

[voodooless]

In the context we are discussing, they absolutely are about preference.

Is it? Why? The first step is establishing that there is an actual audible difference at all. After that is established we can go into why that is, and if one likes one better than another.

Most of the noisy ASR congregation have never actually bothered with ABX testing of anything- they just yell 'ABX!' from the side-lines. It's pathetic, boorish and achieves nothing except make this place look even more fringe, out of touch and full of bat-chit crazy nutcases.

And even if they did bother, what are they actually attempting to achieve in the first place? Mostly, they go in with a pre-set agenda. They want to hear no difference, because then they can justify their cheap-ass purchases of poorly made gear that fails prematurely. That is the bottom line.

I guess that is a matter of "pot calling the kettle black"...

Which brand gets the most air-time on ASR? Topping. Why? Because it kinda tests OK, is cheap, and a noisy bunch of people with clearly limited budgets tell them to buy it to be part of the club. Yawn.

What does any of this matter to the subject at hand?

And, which is the ONLY brand with thread after thread of problems, issues, failures, dissatisfaction and burnt customers? Topping again.

I can repeat myself, but I won't.

Buying and owning HiFi equipment is always about preference. It has never been anything else. ABX is just about whether you can perceive a difference, not which one, two or ten products you prefer over others.

Obviously

It's not the gold standard for selecting HiFi components.

Who says it is? This is just a strawman argument.

Never was and never will be. If it was, there would have been ABX comparators in every HiFi store. But there were comparators in every proper HiFi store because it was easy to hear differences between amplifiers, receivers, tuners, turntables and particularly speakers. Then customers could buy based on their actual preferences. After all, that what is was all about.

It's perceived preference, colored by all kinds of biases that are unknown to people buying these products. I see loads of issues with that.

I choose preference. That encompasses, features, measured performance, build quality, reliability, functionality and plain good engineering.

Sure, you may. But that is not the topic at hand, is it?

Can you hear the difference between the 3 test files?

 

[antcollinet]

I believe I hear differences between the dCS and the other DACs. SINAD measurements provide no insights. RMS null differences with 1kHz sine tone are similar for all DACs.

dBA null with real music alludes to perhaps there being differences between DACs. PK Metric with real music suggests clear differences, therefore the gold standard.

But with no real research into what levels of 'PK metric" are audible. Only your one anecdotal* result. This is not enough to describe it as gold standard. By definition 'standard' has to be standardised, repeatable, verifiable.


*sorry - but without a blind confirmation that you can hear the difference you describe, that is all it is.

 

[voodooless]

dBA null with real music alludes to perhaps there being differences between DACs. PK Metric with real music suggests clear differences, therefore the gold standard.

Based on a sample of 1? Really?