• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Listening Tests - Benchmark DAC3 versus Topping D90LE

OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
But isn't that just the same result with the words changed? What did you mean by switch position "off" in the original representation.

How does the second table change that?

A correct score in this test was simply allowing the blinded person to switch until I thought I was listening to the DAC3.

The question becomes, is the current switch setting the DAC3?
Correct
means the DAC3 was the source, and it was properly identified as the DAC3.
Incorrect means the DAC3 was not the source, and it was improperly in identified as the source.

This is the methodology I chose but it has no impact on the result.
Either I can or cannot and in this small sample, I could 66.7% of the time and could not 33.2% of the time.

- Rich
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,523
Likes
37,056
A correct score in this test was simply allowing the blinded person to switch until I thought I was listening to the DAC3.

The question becomes, is the current switch setting the DAC3?
Correct
means the DAC3 was the source, and it was properly identified as the DAC3.
Incorrect means the DAC3 was not the source, and it was improperly in identified as the source.

This is the methodology I chose but it has no impact on the result.
Either I can or cannot and in this small sample, I could 66.7% of the time and could not 33.2% of the time.

- Rich
So in this testing you heard both dacs each time, and chose which position you thought was the Dac3. Is this correct? That makes more sense to me.
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
So in this testing you heard both dacs each time, and chose which position you thought was the Dac3. Is this correct? That makes more sense to me.
Yes, this is correct. It was simpler that way.

- Rich
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,523
Likes
37,056
Yes, this is correct. It was simpler that way.

- Rich
I think that erases the issues people thought were in place for your test. Level matched, you listened to both at each trial and attempted to pick the DAC 3. Being correct 2/3 of the time.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
Yes, this is correct. It was simpler that way.

- Rich
That was how I interpreted your test. It seemed simple and logical.

Really the only real criticism I could see anyone making is that the sample size was too small to be conclusive, but I understand why you would stop after 15 changes.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
At least it's a test,even flawed we must encourage people,everyone improves with practice.
Again, I thank Rich for taking this on.

At some point I will do a similar test and unless I "prove" that there is no difference, I am sure I will be met with similar grilling and condemnation for lack of proper controls and test procedures... and I am sure there will be issues with my technique that I may be able to improve upon. ;)

Currently what I am interested in comparing are early 80s digital devices against modern DACs.
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,409
Likes
12,294
Location
UK/Cheshire
Again, I thank Rich for taking this on.

At some point I will do a similar test and unless I "prove" that there is no difference, I am sure I will be met with similar grilling and condemnation for lack of proper controls and test procedures... and I am sure there will be issues with my technique that I may be able to improve upon. ;)

Currently what I am interested in comparing are early 80s digital devices against modern DACs.

(EDIT - in saying the below: I very much apprecieate the effort done to carry out this test. It is what people here always say should be done, and controls were carried out pretty well - especially level matching)

Most of the "grilling" (if you want to call it that) has come from the claim that the results "are meaningful", plus some confusion caused by the way the results were initiallly presented.

The 10/15 result is insufficient to show that. In fact there is a 15+% chance of getting that result by pure guessing (or flipping a coin). It is necessary to understand not only how to carry out a test with sufficient controlls, but also how to interpret the results. By the way, the sample size is fine, but it needs to get to 12 correct responses to be better than 95% confidence (the normally expected level) it is not guessing (11 would give 94% - but even then you could 'coin flip' that answer 6 percent of the time)


PS "grilling" is part of the scentific process - in validation of actual science, for example with published papers, it is called "peer review"
 
Last edited:

MaxBuck

Major Contributor
Forum Donor
Joined
May 22, 2021
Messages
1,515
Likes
2,117
Location
SoCal, Baby!
All this discussion really proves is that the number of people who understand proper statistical experiment design is vanishingly small. (And full disclosure: I'm no longer one of them.)
 

IAtaman

Major Contributor
Forum Donor
Joined
Mar 29, 2021
Messages
2,272
Likes
3,836
Very interesting..

Sounds like you think there can be audible differences between "transparent" measuring DACs, yet you were willing to test your theory with some home brewed blind test and even took feedback from the forum on how to improve it. Seems like you are at least questioning your beliefs and I think that is admirable. Maybe the test does not prove anything this way or that way, but as long as the objective of a discussion is for people to understand each other's perspective better, I believe this experiment was a big success. Thank you for taking the time to do the test, and for sharing.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
(EDIT - in saying the below: I very much apprecieate the effort done to carry out this test. It is what people here always say should be done, and controls were carried out pretty well - especially level matching)

Most of the "grilling" (if you want to call it that) has come from the claim that the results "are meaningful", plus some confusion caused by the way the results were initiallly presented.

The 10/15 result is insufficient to show that. In fact there is a 15+% chance of getting that result by pure guessing (or flipping a coin). It is necessary to understand not only how to carry out a test with sufficient controlls, but also how to interpret the results. By the way, the sample size is fine, but it needs to get to 12 correct responses to be better than 95% confidence (the normally expected level) it is not guessing (11 would give 94% - but even then you could 'coin flip' that answer 6 percent of the time)


PS "grilling" is part of the scentific process - in validation of actual science, for example with published papers, it is called "peer review"
I guess my beef is that this test isn't meaningless. While it certainly doesn't "prove" anything it suggests more tests are called for rather than, "nothing to see here."

The attitude from many of the more vocal on ASR is that this topic is "solved" and any exploration is a waste of time. I find that lack of curiosity surprising.

To me the fact that worldwide 10s of thousands of people have an alternate view of this, makes a deeper serious study worthwhile. The results of such a study may very well be that we can not distinguish between the mid level and the high end DACs, however this test while not conclusive, certainly leaves open the possibility that there may be sonic differences between the two devices under test.
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
Thanks to all participating in this thread, including the skeptics.

I believe that skepticism is good. I come at this from the consumer side.
How does a consumer decide which products to purchase. ASR adds tremendous value by performing a set of objective measurements.

DACs are among the least likely to sound differently, but still worthy of investigation.

Processors are completely different animal. The current testing is a good baseline in their performance without processing.
There is little data performance with basic processing engaged. As seen with the MiniDSP, there can be issues introduced in sample-rate conversion that will occur in the likely use case, engaging DSP processing. Any DAC/Processor requires additional basic testing for a baseline of their intended use.

- Rich
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
I guess my beef is that this test isn't meaningless. While it certainly doesn't "prove" anything it suggests more tests are called for rather than, "nothing to see here."

The attitude from many of the more vocal on ASR is that this topic is "solved" and any exploration is a waste of time. I find that lack of curiosity surprising.

To me the fact that worldwide 10s of thousands of people have an alternate view of this, makes a deeper serious study worthwhile. The results of such a study may very well be that we can not distinguish between the mid level and the high end DACs, however this test while not conclusive, certainly leaves open the possibility that there may be sonic differences between the two devices under test.

I still have a MiniDSP SHD (DAC/Processor) to do another session.
Here is a list (thus far) of changes to the methodology under consideration for another blind test:

- Write a program to administer the blind test and record results, reducing listener pressure
- Put the DAC3 in optimal mode - use the default high gain settings instead of the 4 volt
- Use the DAC3 volume control instead of the Roon DSP
- Improve the test result presentation since the original method caused confusion
- Take breaks during the test, to reset your ears and fight fatigue and anxiety
- Perform additional test runs over multiple days

I think the level matching, blinding, automated fast switching all worked well.
The methodology is closer to the Harman model where listeners where blinded but in control of the source material and switching to determine their votes.
IMO, this better accommodates the listener than a strict ABX test.

- Rich
 

IAtaman

Major Contributor
Forum Donor
Joined
Mar 29, 2021
Messages
2,272
Likes
3,836
I guess my beef is that this test isn't meaningless. While it certainly doesn't "prove" anything it suggests more tests are called for rather than, "nothing to see here."

The attitude from many of the more vocal on ASR is that this topic is "solved" and any exploration is a waste of time. I find that lack of curiosity surprising.

To me the fact that worldwide 10s of thousands of people have an alternate view of this, makes a deeper serious study worthwhile. The results of such a study may very well be that we can not distinguish between the mid level and the high end DACs, however this test while not conclusive, certainly leaves open the possibility that there may be sonic differences between the two devices under test.

I think that would be getting ahead of ourselves. These are still 2 inconclusive experiements, let's not forget the evidence on why transparent measuring DACs do not sound different is quite robust and conclusive. Probably a tad bit more conclusive results would be useful to warrant any sort of serious research in my opinion.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
I think that would be getting ahead of ourselves. These are still 2 inconclusive experiements, let's not forget the evidence on why transparent measuring DACs do not sound different is quite robust and conclusive. Probably a tad bit more conclusive results would be useful to warrant any sort of serious research in my opinion.
Great, where is that conclusive evidence. That is what I am looking for.
 

IAtaman

Major Contributor
Forum Donor
Joined
Mar 29, 2021
Messages
2,272
Likes
3,836
Great, where is that conclusive evidence. That is what I am looking for.
There are quite a lot of DAC reviews available here. Majority of the measurements published in those reviews do try and identify whether DACs color sound they are tasked with reproducing or not. I have a lingering suspicion you might be aware of these reviews so why don't you tell me what is it missing from these tests and measurements in your opinion to make them conclusive.
 

sq225917

Major Contributor
Joined
Jun 23, 2019
Messages
1,360
Likes
1,610
Rich, props for putting yourself to the test and posting the results. It's not an easy thing to do, I did it years ago, with the results being that I don't have magic ears either, boo.....
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
There are quite a lot of DAC reviews available here. Majority of the measurements published in those reviews do try and identify whether DACs color sound they are tasked with reproducing or not. I have a lingering suspicion you might be aware of these reviews so why don't you tell me what is it missing from these tests and measurements in your opinion to make them conclusive.

While I respect measurements and use them myself professionally and as an enthusiast, I haven't see the conclusive correlation between human auditory perception and the measured performance of DACs and other electronics that measure well.

I am not even particularly concerned with 99th percentile human auditory perception. Whenever I ask to see the results of such a test, the ASR response is about proving a negative or some similar dismissive response.

I would like to see a test similar to Harman's work with loudspeakers on the the thresholds of audibility with various electronics. Unfortunately it is costly to run such a test and the industry so far has found it more profitable to sell IEC power cables and $10,000 DACs.
 

Purité Audio

Master Contributor
Industry Insider
Barrowmaster
Forum Donor
Joined
Feb 29, 2016
Messages
9,051
Likes
12,150
Location
London
You can perform those tests yourself, choose a ‘transparent’ dac as the benchmark, purchase a dac with known audible distortion ( ie a lot) , compare them level matched and unsighted, you should be able to pick the distorting unit with 100% predictability, then choose a unit with ‘less’ distortion perform the same test, you will discover the amount and type of distortion audible to yourself.
Keith
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,409
Likes
12,294
Location
UK/Cheshire
I guess my beef is that this test isn't meaningless. While it certainly doesn't "prove" anything it suggests more tests are called for rather than, "nothing to see here."

The attitude from many of the more vocal on ASR is that this topic is "solved" and any exploration is a waste of time. I find that lack of curiosity surprising.

To me the fact that worldwide 10s of thousands of people have an alternate view of this, makes a deeper serious study worthwhile. The results of such a study may very well be that we can not distinguish between the mid level and the high end DACs, however this test while not conclusive, certainly leaves open the possibility that there may be sonic differences between the two devices under test.
There are 10's of thousands of people out there that believe the earth is flat, or the moon landings were faked. We don't start serious studies to investigate these things.

We have measurements of this type of gear (Especially DACS) that shows that very many perform such that noise and distortion are both below the level that can be heard by the human ear - also with flat frequency response.

So far no-one has been able to demonstrate clearly that they can hear a difference between devices that measure so well when using similarly competent output filters (EG not selecting slow output filters that roll off the high frequencies)

We also know that everyone is subject to cognitive biases that are more than able to explain unlikely audible differences between identically measuring kit - and can therefore explain 10's of 1000's of people who think they are hearing "night and day" differences.

When someone can conclusively and repeatedly demonstrate audibility of something we are not measuring, or that the measurements don't show - perhaps then some more investigation is warranted.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
You can perform those tests yourself, choose a ‘transparent’ dac as the benchmark, purchase a dac with known audible distortion ( ie a lot) , compare them level matched and unsighted, you should be able to pick the distorting unit with 100% predictability, then choose a unit with ‘less’ distortion perform the same test, you will discover the amount and type of distortion audible to yourself.
Keith
Agreed and plan on doing it when the queue of more important things is gone through.

There are 10's of thousands of people out there that believe the earth is flat, or the moon landings were faked. We don't start serious studies to investigate these things.

We have measurements of this type of gear (Especially DACS) that shows that very many perform such that noise and distortion are both below the level that can be heard by the human ear - also with flat frequency response.

So far no-one has been able to demonstrate clearly that they can hear a difference between devices that measure so well when using similarly competent output filters (EG not selecting slow output filters that roll off the high frequencies)

We also know that everyone is subject to cognitive biases that are more than able to explain unlikely audible differences between identically measuring kit - and can therefore explain 10's of 1000's of people who think they are hearing "night and day" differences.

When someone can conclusively and repeatedly demonstrate audibility of something we are not measuring, or that the measurements don't show - perhaps then some more investigation is warranted.
No argument with any of that.

I have no beef with your logic, but I have not yet seen the conclusive test. Harman's speaker tests are probably not going to convince some "flat earthers" that scientifically designed loudspeakers are preferred by more listeners than magically designed speakers, but I do accept their test results and testing methods. I would like to see this sort of test of these measurably "transparent" electronics before claiming certainty.

To me there is still reasonable doubt.
 
Top Bottom