Master Thread: Are measurements Everything or Nothing?

[storing]

The psychology in China are not the same in the USA.

That's a bit of a stretch. Cultures differ, but principles like confirmation bias, which is studied in the field of psychology, are in fact widely universal (probably even not only in humans but also higher apes, I'd guess). And still science.

 

[voodooless]

That's a bit of a stretch. Cultures differ, but principles like confirmation bias, which is studied in the field of psychology, are in fact widely universal (probably even not only in humans but also higher apes, I'd guess). And still science.

I bet it's mostly the application of that science that is the major difference. Or rather, who applies it: in China it's the government, in the west it's Big Tech

 

[MaxBuck]

Talking wrong and wrong is a universally widespread bias.
In French we said "parlez à tord et à travers".
The English is a language poor.

The psychology in China are not the same in the USA.
The general relativity is the same in China and in the USA.

How is it possible that a so-called scientific forum is such a weak scientific culture.

Je n'sais quoi.

 

[MasterApex]

I have a scale for how much measurements matter for each category of products:

DACs: 100%
Amplifiers (headphone and speaker): 80 to 90% due to variability of available power. Hard to internalize how much power is available/enough without listening tests.
Speakers: 70 to 80%
Headphones: 50 to 80% (measurements too variable)

This is why you see me do listening tests for the last two categories and half of second (headphone amps).

I like this summary and think it explains a lot about speakers
Since not every person ears has the same dBA weighted frequency response, what is neutral on measurement may not be neutral to someone ears.
So the directivity and response plot are good indicators but not the overall story.

However, our ears typically can distinguish "life-like" sound reproduction , meaning the ability to distinguish whether that sound is coming from speakers or a real life trumpet, violin, piano, real person talking, etc.

This is probably why many people like the sound reproduction of B&W 800D3 over competitive Revel, Dynaudio, Magico because the B&W D3 midrange and tweeter can reproduce more "life-like" sounds of closed mic recording. The trumpet sound has the bite like real trumpet playing in front of you.

 

[mrmojo2022]

If you go back, you can see, I posted graphs for 44.1k alt setting 1, 352.8k alt setting 1 and DSD256 alt setting 3 (PCM rate 352800). Mentioned in the graph headers.
View attachment 181245
View attachment 181246
View attachment 181247

Of those, 44.1k PCM was worst performer, 352.8k PCM was a bit better but had picketed noise floor, and DSD256 was cleanest but still abnormal noise floor modulation.

In addition I think I posted graph showing how 19+20 IMD test tone at 705.6k PCM trips up the output.
View attachment 181248

PCM input is 24-bit TPDF dithered data.

@Miska

having just purchased a D90SE I noticed this thread. I thought your results were a bit odd so decided to take my own measurements. All the below 24bit PCM.

No problems are observable and performance is superb.

Unless your D90SE is faulty, and I doubt that is the case, what you are observing is due to your test set up or measurement system (Prism?)

Your noise floor is around -120dB in the FFT. I can only guess at the FFT resolution but that means due to FFT gain the noise floor 20Hz to 20kHz is extremely high, probably less than 100dB which indicates only 16bit acquisition. Assuming this is not a product of the applied test signal, then it indicates the measurement system isnt close to being adequate for the job.

 

[mrmojo2022]

That graph right there shows a problem with the test setup. That's exactly what you see when a scan is interrupted, a connection is intermittent, or an amplitude changes during acquisition. It could also be something in the software chain. Regardless, you need to get your measurements right.

I agree

 

[mrmojo2022]

Hmmh yeah, maybe Topping D90SE measures nice here, but, it depends how you look at things.

I did run some measurements too.

View attachment 180265

OK, that looked decent, let's see how it looks like with -60 dBFS input:
View attachment 180266

OK, WTF is going on here? Let's try the same -60 dBFS signal with 352.8k rate:
View attachment 180267

OK, better, but something is still wrong. How about DSD256 input:
View attachment 180268

Better still, but still some noise floor modulation issues.

If we then take a half price DAC with same test, same DAC chip:
View attachment 180273


View attachment 180269

Now that's better!

And we can even go to about 250€ PCIe card based on AKM chip:
View attachment 180272

So maybe choosing a DAC based on the SINAD chart is not the best thing to do. I wasted 900€, and the D90SE stays in the box. I had totally opposite experience with the ADI-2 for example. Maybe more thorough testing by the manufacturer?

Sure your prism hasn't got a noisy pre amp that auto ranged and slapped a bunch of gain on the -60dB signal?

 

[mrmojo2022]

I don't think the Prism system is up to the job. 105dB thd+n

 

[Miska]

No problems are observable and performance is superb.

Unless your D90SE is faulty, and I doubt that is the case, what you are observing is due to your test set up or measurement system (Prism?)

First question, which one of the two PCM USB endpoints are you using in your tests? D90SE has two PCM altsettings and one DSD altsetting. I never use Windows drivers or such, source is always Linux with the built-in kernel UAC2 driver.

At 705.6k, do a linear sweep and observe the behavior. Here's 0 - 22.05 kHz linear sweep, at 705.6k sampling rate and analyzer in peak-hold mode.

It won't happen at 1 kHz tone for this sweep, but once the sweep reaches about 10 kHz it goes nuts. This also happens for DSD inputs at both DSD256 and DSD512, even with 1 kHz tones.

Low level sweep at 705.6k rate:

I have now tested this on two physically distinct locations and three different measurement systems. Nothing else in common except the D90SE.

I don't think the Prism system is up to the job. 105dB thd+n

That is what they promise, but my unit performs much better than that.

For comparison, EVGA NU-Audio PCIe sound card, -60 dBFS 1 kHz sine at DSD256 input:

Or Holo Audio Spring 3, 1 kHz at -120 dB 20-bit noise-shaped input at 705.6k:

what you are observing is due to your test set up or measurement system

Interesting it happens only with this DAC, out of ~30 others. Maybe my unit is broken, but if others also measure changes in noise floor with DSD inputs and worsening THD+N as rate increases, there's something wrong with the DAC, because the ESS Sabre doesn't have such behavior.

For example SMSL M500 mkII:

 

[mrmojo2022]

First question, which one of the two PCM USB endpoints are you using in your tests? D90SE has two PCM altsettings and one DSD altsetting. I never use Windows drivers or such, source is always Linux with the built-in kernel UAC2 driver.

At 705.6k, do a linear sweep and observe the behavior. Here's 0 - 22.05 kHz linear sweep, at 705.6k sampling rate and analyzer in peak-hold mode.
View attachment 187214

It won't happen at 1 kHz tone for this sweep, but once the sweep reaches about 10 kHz it goes nuts. This also happens for DSD inputs at both DSD256 and DSD512, even with 1 kHz tones.

Low level sweep at 705.6k rate:
View attachment 187215

I have now tested this on two physically distinct locations and three different measurement systems. Nothing else in common except the D90SE.



That is what they promise, but my unit performs much better than that.

For comparison, EVGA NU-Audio PCIe sound card, -60 dBFS 1 kHz sine at DSD256 input:
View attachment 187216

Or Holo Audio Spring 3, 1 kHz at -120 dB 20-bit noise-shaped input at 705.6k:
View attachment 187217


Interesting it happens only with this DAC, out of ~30 others. Maybe my unit is broken, but if others also measure changes in noise floor with DSD inputs and worsening THD+N as rate increases, there's something wrong with the DAC, because the ESS Sabre doesn't have such behavior.

For example SMSL M500 mkII:
View attachment 187220

View attachment 187221

Usb end points? no idea what you are referring to. Simply plugged into a Windows laptop via USB and using REW.

Clearly your Linux test setup is broken. Can you explain why your noise floor is up at -120dB? So we can work out what the FFT gain is, can you tell us what the FFT resolution is, or better still get the prism software to show a 20Hz to 20kHz overall noise level?

Before you start asserting that there are issues in the MHz region, shall we first establish if you are able to accurately measure in the audible region?

Well maybe the dac is broken, but I think its highly unlikely and TBH those results should have triggered you to look elsewhere for set up issues, or confirmation from other peoples tests before suggesting an issue with the design.

Again, with your additional plots you need to provide proper test set up info. what fft resolution or overall noise level 20 to 20khz. it's obvious the fft resolution is cranked right up to get the Prism to show a noise floor at -160dB.

 

[pkane]

First question, which one of the two PCM USB endpoints are you using in your tests? D90SE has two PCM altsettings and one DSD altsetting. I never use Windows drivers or such, source is always Linux with the built-in kernel UAC2 driver.

At 705.6k, do a linear sweep and observe the behavior. Here's 0 - 22.05 kHz linear sweep, at 705.6k sampling rate and analyzer in peak-hold mode.
View attachment 187214

It won't happen at 1 kHz tone for this sweep, but once the sweep reaches about 10 kHz it goes nuts. This also happens for DSD inputs at both DSD256 and DSD512, even with 1 kHz tones.

Low level sweep at 705.6k rate:
View attachment 187215

I have now tested this on two physically distinct locations and three different measurement systems. Nothing else in common except the D90SE.



That is what they promise, but my unit performs much better than that.

For comparison, EVGA NU-Audio PCIe sound card, -60 dBFS 1 kHz sine at DSD256 input:
View attachment 187216

Or Holo Audio Spring 3, 1 kHz at -120 dB 20-bit noise-shaped input at 705.6k:
View attachment 187217


Interesting it happens only with this DAC, out of ~30 others. Maybe my unit is broken, but if others also measure changes in noise floor with DSD inputs and worsening THD+N as rate increases, there's something wrong with the DAC, because the ESS Sabre doesn't have such behavior.

For example SMSL M500 mkII:
View attachment 187220

View attachment 187221

@Miska , let's be very clear here. Your original post stated that this DAC has a problem specifically at 44.1kHz sampling rate at -60dBFS. Your measurement with a single 1kHz tone was what was in question:

Measurements of other DACs, sweeps, etc., are interesting, but not very relevant here. Your 1kHz -60dB measurement is. So, how do you reconcile your measurement with that by @mrmojo2022 which shows none of the errors seen in yours?

 

[Miska]

@Miska , let's be very clear here. Your original post stated that this DAC has a problem specifically at 44.1kHz sampling rate at -60dBFS. Your measurement with a single 1kHz tone was what was in question:

Measurements of other DACs, sweeps, etc., are interesting, but not very relevant here. Your 1kHz -60dB measurement is. So, how do you reconcile your measurement with that by @mrmojo2022 which shows none of the errors seen in yours?

Those sweeps are interesting for figuring out where all the noise floor dirt comes from. And I think I found the explanation.

I suspect the difference may be due to different kinds of anti-alias filters, given that the DAC spits out so much high frequency energy in MHz range. I should have probably plugged in the Prism class-D amplifier pre-filter device for this one.

Following plots are with continuous averaging.

If we focus on those -60 dB 1 kHz tones. Here's 44.1k:

And same but 352.8k input rate:

And then at 705.6k:

 

[Miska]

Usb end points? no idea what you are referring to. Simply plugged into a Windows laptop via USB and using REW.

You know how USB Audio Class works?

Clearly your Linux test setup is broken.

OK, again my test setup is consistently broken only for one DAC out of about 30 others?

Can you explain why your noise floor is up at -120dB? So we can work out what the FFT gain is, can you tell us what the FFT resolution is, or better still get the prism software to show a 20Hz to 20kHz overall noise level?

Level of the noise floor is not relevant. What is relevant is that it is not clean but has discrete dirt.

Before you start asserting that there are issues in the MHz region, shall we first establish if you are able to accurately measure in the audible region?

I think I have shown with other DACs that I can?

Well maybe the dac is broken, but I think its highly unlikely and TBH those results should have triggered you to look elsewhere for set up issues, or confirmation from other peoples tests before suggesting an issue with the design.

Yes, so I have spent so far two full days measuring that particular DAC. On three different setups.

 

[Miska]

There can be also issues after changing between different modes, PCM, DSD and different rates.

My 44.1k results with Prism are after measuring with DSD512, DSD256, 705.6k PCM and 352.8k PCM.

Here's an example of bug in SMSL M500 mkII, if you want to use 705.6/768k rates, those must be the first and only rates you use then it works fine:

If you switch to 352.8k rate and then back to 705.6k, without rebooting the DAC, this happens:

That is what I'd call a buggy firmware.

 

[dc655321]

There can be also issues after changing between different modes, PCM, DSD and different rates.

My 44.1k results with Prism are after measuring with DSD512, DSD256, 705.6k PCM and 352.8k PCM.

Here's an example of bug in SMSL M500 mkII, if you want to use 705.6/768k rates, those must be the first and only rates you use then it works fine:
View attachment 187271

If you switch to 352.8k rate and then back to 705.6k, without rebooting the DAC, this happens:
View attachment 187272

That is what I'd call a buggy firmware.

When you find an issue like that, do you reach out to the manufacturer?

 

[Miska]

When you find an issue like that, do you reach out to the manufacturer?

Depends if they are reachable and have been talking to me. If they want to provide devices for me for compatibility testing then yes.

In this case I saw D90SE test results by some other people that didn't match my expectations of device with this DAC chip. So I purchased one for testing to see if it was a problem in the new E1DA Cosmos ADC used for those testing or problem with the DAC. I suspected former, but found out that it was the latter one. The ADC was fine and the DAC was behaving strange.

 

[mrmojo2022]

Those sweeps are interesting for figuring out where all the noise floor dirt comes from. And I think I found the explanation.

I suspect the difference may be due to different kinds of anti-alias filters, given that the DAC spits out so much high frequency energy in MHz range. I should have probably plugged in the Prism class-D amplifier pre-filter device for this one.

Following plots are with continuous averaging.

If we focus on those -60 dB 1 kHz tones. Here's 44.1k:
View attachment 187266

And same but 352.8k input rate:
View attachment 187267

And then at 705.6k:
View attachment 187268

So you are now saying it's a test set up problem and your Prism system is a radio receiver.

 

[mrmojo2022]

You know how USB Audio Class works?



OK, again my test setup is consistently broken only for one DAC out of about 30 others?



Level of the noise floor is not relevant. What is relevant is that it is not clean but has discrete dirt.



I think I have shown with other DACs that I can?



Yes, so I have spent so far two full days measuring that particular DAC. On three different setups.

Irrelevant.

You have just admitted the problem is with your test set up.

Level of noise floor is absolutely relevant. your system is measuring it incorrectly.

No you have shown you didn't recognise an obvious measurement issue. multiple others did, so excuse us if we distrust your conclusions.

Seems like that was a lot of wasted time.

 

[mrmojo2022]

Depends if they are reachable and have been talking to me. If they want to provide devices for me for compatibility testing then yes.

In this case I saw D90SE test results by some other people that didn't match my expectations of device with this DAC chip. So I purchased one for testing to see if it was a problem in the new E1DA Cosmos ADC used for those testing or problem with the DAC. I suspected former, but found out that it was the latter one. The ADC was fine and the DAC was behaving strange.

You actually found it was a test setup problem not a dac problem.

 

[Blumlein 88]

Well we wanted someone with a D90 to do a confirming test. Test was done, and no confirmation that -60 or -63 db signal at 44.1 khz has a problem. So looks like a test setup or condition issue or a faulty unit.

So Miska do you still get a 1 khz -60 db tone result like you've shown earlier and how do you explain it vs mrmojo2022's result?