• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Listening Tests - Benchmark DAC3 versus Topping D90LE

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
How many times do you think you might get it right (on average) if you were purely guessing. What do you think the possible test to test variation on that might be?

Also - the listener response was always "off" which suggests you were not hearing any difference at all.

Respectfully, we will have to agree to disagree on the usefulness of this test.

I for one would not take these odds as to whether or not the chamber of a gun pointed at my head was loaded, but for something as life critical as audio, I think this is a valid data point if not a definitive result.
 

Owl

Member
Joined
Jun 7, 2022
Messages
77
Likes
118
How would you level match the two units? Output from the device or volume control?
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
How many times do you think you might get it right (on average) if you were purely guessing. What do you think the possible test to test variation on that might be?

Also - the listener response was always "off" which suggests you were not hearing any difference at all.

Easy 50%.
Actually no. OFF refers to the state of the Shelly1 Relay state on the web app.
When OFF the ARX RS-1 receives no power, selecting the default input, in this case the DAC3.
When ON 12 volts are sent to the ARX RS-1 to select the seconds input, in this case was the D90LE.

I stopped on OFF each time because that was what I thought was the DAC3.
That was my choice, I was right 66.7 percent and wrong 33.3 percent of the time.

- Rich
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
How would you level match the two units? Output from the device or volume control?

Though the DAC3 has volume control, I chose to set it in HT mode.
Output was measured at the amplifier output while driving the speakers using a multimeter as described below.

Level Matching
The levels were matched using the Roon DSP volume control and measuring the voltage (using a FLUKE 87V true sinewave multimeter) while playing a 1 kHz 0dBFS tone into the M126bes.
The target voltage is 2.83 volts, commonly used since at 8 Ohms this is 1 watt. This was also the listening level used.

D90 2.807 volts: D90VoltageDSP-14.jpg DAC3 2.825: DAC3VoltageDSP-12.jpg
The DAC3 has a slightly more output of 0.018 volts.

I am no EE, but I supposed that voltage is linear to dB, so I measured the DAC3 adjusting the volume down 1 db.
The 1 dB difference reduced the voltage by 0.244, so (if this calculation makes sense) the volume difference is 0.07 dB by voltage, well below the commonly recommended 0.1 dB level match.

- Rich
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,408
Likes
12,291
Location
UK/Cheshire
Respectfully, we will have to agree to disagree on the usefulness of this test.

I for one would not take these odds as to whether or not the chamber of a gun pointed at my head was loaded, but for something as life critical as audio, I think this is a valid data point if not a definitive result.
But if you were guessing, you'd expect - on average to get 50% guesses right.

If you tossed a coin 15 times, on average you'd get 7 or 8 heads (you can't get 7.5). But you wouldn't get 7 or 8 every time you flipped 15 times : that would be really strange. Would you really be surprised if you got 10 heads sometimes - or even 11 or 12?

I just set up a spreadsheet to do 15 coin tosses 13 times. I got 4 10's, and one 12. Only 5 were the average 7/8. Your test is statistically invalid. (Little more significant than a random guess)
 

MattHooper

Master Contributor
Forum Donor
Joined
Jan 27, 2019
Messages
7,194
Likes
11,806
The blind test is very meaningful - it proved you could not tell the difference between the two DACs - as expected.

On one hand your comment springs from an appropriate level of statistical caution.

On the other, it seems you've stated an inappropriately strong conclusion from RichB's test (for similar reasons to why you are challenging RichB's interpretation in the first place).
 

Martin

Major Contributor
Forum Donor
Joined
Mar 23, 2018
Messages
1,895
Likes
5,536
Location
Cape Coral, FL
Your
I suppose we find the answer we are looking for.

- Rich

You’re kidding right? You guessed the DAC3 was playing for every trial. You only got 66.7% right because the DAC3 was actually playing 10 out of 15 times. It was worse than guessing because you never said the D90ES was playing. So of course you’ll get it right when the DAC3 is playing. Try again only actually guess one or the other. And have it be the DAC3 half the time and the D90SE half the time and see how you do. If you guess DAC3 under that test you’ll only score 50%. This is a perfect example of how not to do a blind test.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
Though the DAC3 has volume control, I chose to set it in HT mode.
Output was measured at the amplifier output while driving the speakers using a multimeter as described below.

Level Matching
The levels were matched using the Roon DSP volume control and measuring the voltage (using a FLUKE 87V true sinewave multimeter) while playing a 1 kHz 0dBFS tone into the M126bes.
The target voltage is 2.83 volts, commonly used since at 8 Ohms this is 1 watt. This was also the listening level used.

D90 2.807 volts: D90VoltageDSP-14.jpg DAC3 2.825: DAC3VoltageDSP-12.jpg
The DAC3 has a slightly more output of 0.018 volts.

I am no EE, but I supposed that voltage is linear to dB, so I measured the DAC3 adjusting the volume down 1 db.
The 1 dB difference reduced the voltage by 0.244, so (if this calculation makes sense) the volume difference is 0.07 dB by voltage, well below the commonly recommended 0.1 dB level match.

- Rich
That level matching is fine, but I thought you previously said you matched using pink noise and a sound meter?

As for validity of the test, you need 12 of 15 to meet the normal 5% chance it is random level. Well actually barely over 11. In any case fell just short. Maybe repeat it another time and you'll have better data.

I find if you've done a few blind listening sessions, they aren't stressful. Tedious, but not stressful. It isn't a test of your hearing ability or your audiophile cred. It is a test to see if two bits of gear are audible or not. And bottom line to any of this, if you feel better with the more expensive DAC, then keep it and you'll not be worried.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
And maybe you really do. It is extremely difficult to setup a blind test properly, in case of tiny differences. They may be erased in the test. On the other hand, the difference is probably unimportant.
I agree both ways about tiny differences and the unimportance. I've always considered if you can tell something 17 or more times out of 20 it is large enough it might matter, and certainly when you can get 20 of 20. OTOH, 548 out of 1000 trials has a p-value of less than 1 %. If you are talking about something that small it isn't a big difference. Even something for 5% difference with 100 trials being 60 out of 100 has gotten to be a small difference. Smaller than how you feel one day versus another.
 

SorenTyson

Member
Joined
Feb 11, 2023
Messages
43
Likes
66
Easy 50%.
Actually no. OFF refers to the state of the Shelly1 Relay state on the web app.
When OFF the ARX RS-1 receives no power, selecting the default input, in this case the DAC3.
When ON 12 volts are sent to the ARX RS-1 to select the seconds input, in this case was the D90LE.

I stopped on OFF each time because that was what I thought was the DAC3.
That was my choice, I was right 66.7 percent and wrong 33.3 percent of the time.

- Rich

I wonder, if you picked the same every time, that must mean that you heard no difference (if so you should have changed to on at some point?)

The percentage right is just totally random based on what your daughter selected. She should have picked 50/50.
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
Your


You’re kidding right? You guessed the DAC3 was playing for every trial. You only got 66.7% right because the DAC3 was actually playing 10 out of 15 times. It was worse than guessing because you never said the D90ES was playing. So of course you’ll get it right when the DAC3 is playing. Try again only actually guess one or the other. And have it be the DAC3 half the time and the D90SE half the time and see how you do. If you guess DAC3 under that test you’ll only score 50%. This is a perfect example of how not to do a blind test.

I chose to only try to guess only when the DAC3 was playing as a methodology.

- Starting point was randomized
- Switching, track selection, and decision point controlled by the listener
- Run selection was terminated when the source was thought to be DAC3

Here is the same data simplified:

DAC3 v D90LE blind test Simplified.jpg

The choice to select on the DAC3 as source had no impact on the odds.
It is the same as coin flipping and always select tails, it has no impact on the odds.

I suppose there are those who believe if the bet on red on roulette, you are influencing the outcome. :)

- Rich
 

Martin

Major Contributor
Forum Donor
Joined
Mar 23, 2018
Messages
1,895
Likes
5,536
Location
Cape Coral, FL
I chose to only try to guess only when the DAC3 was playing as a methodology.

- Starting point was randomized
- Switching, track selection, and decision point controlled by the listener
- Run selection was terminated when the source was thought to be DAC3

Here is the same data simplified:

View attachment 269603
The choice to select on the DAC3 as source had no impact on the odds.
It is the same as coin flipping and always select tails, it has no impact on the odds.

I suppose there are those who believe if the bet on red on roulette, you are influencing the outcome. :)

- Rich

Exactly. It’s not a real test. The only thing your methodology showed was that you couldn’t hear a difference. If it was a valid test you would have proven that to yourself.
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
Exactly. It’s not a real test. The only thing your methodology showed was that you couldn’t hear a difference. If it was a valid test you would have proven that to yourself.

At no time, was I confused about your position in this matter.
Sure, this is not statistically significant, I did not say it was, but I have no predetermined bias.
Knowing what the results would be, that is the definition of a predetermined bias.

Your


You’re kidding right? You guessed the DAC3 was playing for every trial. You only got 66.7% right because the DAC3 was actually playing 10 out of 15 times. It was worse than guessing because you never said the D90ES was playing. So of course you’ll get it right when the DAC3 is playing. Try again only actually guess one or the other. And have it be the DAC3 half the time and the D90SE half the time and see how you do. If you guess DAC3 under that test you’ll only score 50%. This is a perfect example of how not to do a blind test.

It was this statement that makes me question your understanding of how odds and statistics work.
With two choices, implicit the selection of one means, it is not the other.
There is no bearing on likely outcome. This is illogical.

If you're position is that this proved nothing, I agree.
If you're position is that I did something wrong, I'm listening.

IMO, The test was adequately blinded, it was level matched to less-than 0.1 dB, it had fast switching, and it put the listening in control of the tracks and time to select.
Nothing is perfect, but if you have beef with the setup, then again, I'm listening.

- Rich
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,408
Likes
12,291
Location
UK/Cheshire
At no time, was I confused about your position in this matter.
Sure, this is not statistically significant, I did not say it was, but I have no predetermined bias.
Knowing what the results would be, that is the definition of a predetermined bias.



It was this statement that makes me question your understanding of how odds and statistics work.
With two choices, implicit the selection of one means, it is not the other.
There is no bearing on likely outcome. This is illogical.

If you're position is that this proved nothing, I agree.
If you're position is that I did something wrong, I'm listening.

IMO, The test was adequately blinded, it was level matched to less-than 0.1 dB, it had fast switching, and it put the listening in control of the tracks and time to select.
Nothing is perfect, but if you have beef with the setup, then again, I'm listening.

- Rich
I think what most of us are having a problem with is this:

You apparently guessed the same position of the switch all 15 times. That implies that each test sounded the same to you regardless of which position the switch was in. You only got some "right" because sometimes the switch was in the same postion that you had chosen for all the tests.

Screenshot 2023-03-05 at 21.06.09.png
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,146
Likes
1,697
Location
SF Bay Area
I think what most of us are having a problem with is this:

You apparently guessed the same position of the switch all 15 times. That implies that each test sounded the same to you regardless of which position the switch was in. You only got some "right" because sometimes the switch was in the same postion that you had chosen for all the tests.
I think that Rich's chart and nomenclature were confusing. I believe his simplified version where simply says that his call out was correct or incorrect is the accurate interpretation.

This is still not a large enough sample size or percentage of correct choices to be statistically significant, but he wasn't guessing the same state every time and occasionally was correct.
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,408
Likes
12,291
Location
UK/Cheshire
I think that Rich's chart and nomenclature were confusing. I believe his simplified version where simply says that his call out was correct or incorrect is the accurate interpretation.

This is still not a large enough sample size or percentage of correct choices to be statistically significant, but he wasn't guessing the same state every time and occasionally was correct.
Are you sure? Why would he state the swtich position "listener respose" as being "off" every time?
 

Sokel

Master Contributor
Joined
Sep 8, 2021
Messages
5,834
Likes
5,760
Did this test done for proof of DACs been different just for the shake of it or for preference?
If the first it proved nothing,if the second the conditions should be different.
At least it's a test,even flawed we must encourage people,everyone improves with practice.
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
I think what most of us are having a problem with is this:

You apparently guessed the same position of the switch all 15 times. That implies that each test sounded the same to you regardless of which position the switch was in. You only got some "right" because sometimes the switch was in the same postion that you had chosen for all the tests.

View attachment 269702
Here is a better representation of the same results:

DAC3 v D90LE blind test Simplified.jpg

Each test run is essentially, selecting for the source as the DAC3.

A small sample, to be sure. If I do this again, with a different DAC since the D90LE is in the box.

I may write a program to randomize the starting DAC, blind the test, record the results, and report the data.

- Rich
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,408
Likes
12,291
Location
UK/Cheshire
Here is a better representation of the same results:

View attachment 269714
Each test run is essentially, selecting for the source as the DAC3.

A small sample, to be sure. If I do this again, with a different DAC since the D90LE is in the box.

I may write a program to randomize the starting DAC, blind the test, record the results, and report the data.

- Rich
But isn't that just the same result with the words changed? What did you mean by switch position "off" in the original representation.

How does the second table change that?
 
OP
RichB

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
Are you sure? Why would he state the swtich position "listener respose" as being "off" every time?

I see that this was not the best way to present the data.
Off refers the the state of the Shelly1 relay web app.

Here is a picture of ON, which means the D90LE was playing.

1678053092942.png


Here is a picture of OFF wnich means the DAC3 is playing.

1678053110344.png


I simple chose to try to identify the DAC3 for simplicity.
That choice has no real bearing on the results.
My daughter did no know the result of ON or OFF or what I was testing.
She merely recorded the actual state versus my response.

- Rich
 
Top Bottom