• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

DAC and amp combos did not give same clues when running online blind tests. Why? What would be the desired clue?

OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Excuse me if I have this wrong.

To be valid, apart from being double blinded, you'd have to run each stack multiple times out of sequence. Not attempt it ten times on one stack then move onto the next, you'd have to randomise the stack order and run it myiple times.

As it stands it appears to offer no statistical validity.
If both chains are confirmed voltage matched and I can identify which chain is which in a property managed blind tests, where which chain to be selected is determined by a coin flip, I would say getting 50/50 would satisfy most people here?
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
@dominikz , because of your suggestions, I decided to do voltage matched sighted listening checks and do frequency response captures. I think maybe I found a reason why Gustard combo has no extra hit.

First, I did voltage matched sighted listening check:
The output levels used are the comfortable output level when using he400se to run tests. Topping combo is 0.16V for Right channel and 0.158 for Left channel, when measuring with multimeter, playing a 90hz tone. Gustard combo is 0.16V for Right channel and 0.163 for Left channel, when measuring with multimeter, playing 90hz tone.

When voltage matched, I still sense that Topping combo has that extra hit (that hit is lower freq kind of hit and not high freq). Gustard does not has the extra hit. Even when I up the volume and down the volume of the Gustard combo a notch, I still would not sense that extra low freq hit.


Edit:
Removed the frequecy respose measurements I previously attached below. Something seriously wrong betwen my Focusrite Forte and REW, it would seem. I tried Focustite Scarlett and the charts are not like these. More charts to follow in newer posts.
 
Last edited:
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Here is the freq response as captured with Focusrite Scarlett

The chain:
AMP jack -> jack to RCA cable -> RCA to XLR cable -> XLR input of Forte Scarlett

topping_vs_gustard_chain_20hz_to_20khz_via_Focusrite_Scarlett.PNG


The different in db seems to match what I measured with multimeter at 90hz. It would seems I still can't find explanation why Topping combo gives me extra hit and Gustard combo is not.
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Here is the freq response as captured with Focusrite Forte. Second tries worked once I use default device as input instead of using Forte as input device.

The chain:
AMP jack -> jack to RCA cable -> RCA to XLR cable -> XLR input of Focusrite Forte

topping_vs_gustard_chain_20hz_to_20khz_via_Focusrite_Forte.PNG


It would seem pretty much matching Scarlett (other than extreme low) and similar to multi meter measurements at 90Hz.
 
Last edited:
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Why do you spend so much time on so little?
Because I am still curious why I get different clues with different chain, when running online timing tests. Could the different clues tell me something more, like which chain is best for primary listening chain???

As far as I can see with measurements, there is no frequency response issue that would explain that.

Even when voltage matched as requested, different transparent chain combo still sounds different. This does not match well with "everything should sound the same with transparent chain" camp.

: P
 

Chrispy

Master Contributor
Forum Donor
Joined
Feb 7, 2020
Messages
7,751
Likes
5,910
Location
PNW
The clues may just be telling you at different times/places/efforts you're still being influenced more by your mind than your ears....and it's still trivial.

ps I'd call it obsessive mostly

pps If I ever get up to Portland hope to get a definitive demo :)
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
The clues may just be telling you at different times/places/efforts you're still being influenced more by your mind than your ears....and it's still trivial.

ps I'd call it obsessive mostly

pps If I ever get up to Portland hope to get a definitive demo :)
Like I said, I am willing to do something like a $10k challenge to prove that it is not my mind.

Want Amir to run this challenge and we do this bet? Let's make it something beneficial to Amir too.

If I pass, you pay Amir $5k and me $5k. If I can't pass, I pay Amir $5k and you $5k.

Sound like easy money to you. Want to take this challenge?
 

Chrispy

Master Contributor
Forum Donor
Joined
Feb 7, 2020
Messages
7,751
Likes
5,910
Location
PNW
Like I said, I am willing to do something like a $10k challenge to prove that it is not my mind.

Want Amir to run this challenge and we do this bet? Let's make it something beneficial to Amir too.

If I pass, you pay Amir $5k and me $5k. If I can't pass, I pay Amir $5k and you $5k.

Sound like easy money to you. Want to take this challenge?
I've no interest in such. I doubt you'll win, tho.
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
I've no interest in such. I doubt you'll win, tho.
Fine. This challenge is still open to all "measured transparent means sound the same" camp members.
 

Chrispy

Master Contributor
Forum Donor
Joined
Feb 7, 2020
Messages
7,751
Likes
5,910
Location
PNW
Fine. This challenge is still open to all "measured transparent means sound the same" camp members.
Would love to see this carried out....but the terms/meeting can be difficult to arrange. Then again, why is it even important at all?
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Would love to see this carried out....but the terms/meeting can be difficult to arrange. Then again, why is it even important at all?
See first post?

Again:
Curious if those clues would help me select the best combo as my primary listening chain.
 

dominikz

Addicted to Fun and Learning
Forum Donor
Joined
Oct 10, 2020
Messages
803
Likes
2,626
Even when voltage matched as requested, different transparent chain combo still sounds different. This does not match well with "everything should sound the same with transparent chain" camp.
While I do think it is commendable that you repeated the test with level matching, it is IMO much too early to jump to any conclusions - TBH repeating your claims after each round of informal tests makes it seem like your mind is already set on the desired outcome, even before attempting any test. This is suggestive of confirmation bias, making it more difficult for you to facilitate a reliable test in the first place; and unfortunately also for others to have confidence in the results you present.
Please don't take this personally - we all sometimes fall prey to confirmation bias - that comes with being human. :) However it is why we have to work hard to eliminate its effects from our investigations. It is also prudent to be reasonably skeptical of one's own work - especially when it is producing unexpected results.

Lastly, I hope you realize that listening sighted is a serious problem. You may notice that not all test variants I proposed in post #4 are level-matched, but all are blind. In addition, multiple randomized trials would be required to reach any statistical significance. It can be tedious work. :)

Like I said, I am willing to do something like a $10k challenge to prove that it is not my mind.

Want Amir to run this challenge and we do this bet? Let's make it something beneficial to Amir too.

If I pass, you pay Amir $5k and me $5k. If I can't pass, I pay Amir $5k and you $5k.

Sound like easy money to you. Want to take this challenge?
Honest question - if your motivation is to learn and understand, why offer money / make a challenge out of it at all?
If you're honestly interested in this line of investigation (as you seem to be, given the time invested so far), and believe you have solid findings that corroborate your point of view, why not spend the time, money and effort to instead learn more about controlled listening test methodology yourself, build a solid controlled test protocol, and try to present your findings somewhere for formal peer-review?

Here is the freq response as captured with Focusrite Forte. Second tries worked once I use default device as input instead of using Forte as input device.

The chain:
AMP jack -> jack to RCA cable -> RCA to XLR cable -> XLR input of Focusrite Forte

View attachment 190919

It would seem pretty much matching Scarlett (other than extreme low) and similar to multi meter measurements at 90Hz.
Do you have any explanation for the pretty severe HF roll-off seen in the Topping stack FR (-5dB @ 20kHz)?
'Hairiness' in the FR plots may also suggest some issue with the setup - perhaps driver-related? In addition to FR, I'd suggest to also show a distortion vs frequency plot.
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
While I do think it is commendable that you repeated the test with level matching, it is IMO much too early to jump to any conclusions - TBH repeating your claims after each round of informal tests makes it seem like your mind is already set on the desired outcome, even before attempting any test. This is suggestive of confirmation bias, making it more difficult for you to facilitate a reliable test in the first place; and unfortunately also for others to have confidence in the results you present.
Please don't take this personally - we all sometimes fall prey to confirmation bias - that comes with being human. :) However it is why we have to work hard to eliminate its effects from our investigations. It is also prudent to be reasonably skeptical of one's own work - especially when it is producing unexpected results.

Lastly, I hope you realize that listening sighted is a serious problem. You may notice that not all test variants I proposed in post #4 are level-matched, but all are blind. In addition, multiple randomized trials would be required to reach any statistical significance. It can be tedious work. :)


Honest question - if your motivation is to learn and understand, why offer money / make a challenge out of it at all?
If you're honestly interested in this line of investigation (as you seem to be, given the time invested so far), and believe you have solid findings that corroborate your point of view, why not spend the time, money and effort to instead learn more about controlled listening test methodology yourself, build a solid controlled test protocol, and try to present your findings somewhere for formal peer-review?


Do you have any explanation for the pretty severe HF roll-off seen in the Topping stack FR (-5dB @ 20kHz)?
'Hairiness' in the FR plots may also suggest some issue with the setup - perhaps driver-related? In addition to FR, I'd suggest to also show a distortion vs frequency plot.


That is all I can do for now as my wife already told me yesterday she won't help me with blind tests.
; (

Regarding the Forte, don't worry about it. I think the driver somehow no longer supported correctly with latest revision of Windows 10. Check the plot above it, use the Scarlett one instead.

Regarding the slope for Topping combo, that one is very interesting and would need further check as this can be another support for "not all transparent combo sounds the same". But, I suspect that slope has no effect on my hearing because I can't hear over 15khz anyway. Also, the timing test does not have frequency that high anyway, so not really important for now. The extra hit I sensed is low frequency kind and not high frequency kind.

Now, ask yourself this question:
For an experienced tester who have done all the tests as described in
https://www.audiosciencereview.com/...tracking-my-listening-tests.24364/post-822052 .....
would he still not know what is obvious and what is not obvious?

Would he still not know what kind of tests are easy as pie, what kind of tests are not easy?

Hearing the extra hit clue with Topping combo is easy as pie. There is no imagination bias. Imagination bias won't help anyone pass online blind tests easily.

Instead of keep asking me doing the kind of blind tests that I can't do anytime soon, how about yourself do a few rounds of the online blind tests and see for yourself if you sense similar kind of different clues? You might not hear it, at least I will be very appreciative that you tried to do peer review.

Thanks again!

: )
 
Last edited:

dominikz

Addicted to Fun and Learning
Forum Donor
Joined
Oct 10, 2020
Messages
803
Likes
2,626
Now, ask yourself this question:
For an experienced tester who have done all the tests as described in
https://www.audiosciencereview.com/...tracking-my-listening-tests.24364/post-822052 .....
would he still not know what is obvious and what is not obvious?

Would he still not know what kind of tests are easy as pie, what kind of tests are not easy?
Unfortunately we're all subject to cognitive bias - regardless of our awareness of its existence, or our critical listening competence. Note that here you appear to be appealing to a variation of 'argument from authority', a common fallacy.

Instead of keep arguing with me, how about yourself do a few rounds of the online blind tests and see for yourself if you sense similar kind of different clues? You might not hear it, at least I will be very appreciative that you tried to do peer review.
I'm a bit saddened if you see my posts as 'arguing'. My genuine intention was to provide a possible answer to the question you posed in this thread and to try and help you improve your test methodology.

BTW - me doing the same test as you would not be peer review. Peer review means looking at your methodology and conclusions and providing comments and proposals for improvements (where applicable) - i.e. similar to what I tried to do up to now. :)
What you suggest I do is independently replicate your test and confirm / refute the validity of your results. However for people to be interested in replication you should first be able to show that your method is valid.

Anyway I think there's not much more I can help with so I'll drop out now. I do hope you found some of the information helpful and wish you good luck with your investigations!
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Unfortunately we're all subject to cognitive bias - regardless of our awareness of its existence, or our critical listening competence. Note that here you appear to be appealing to a variation of 'argument from authority', a common fallacy.


I'm a bit saddened if you see my posts as 'arguing'. My genuine intention was to provide a possible answer to the question you posed in this thread and to try and help you improve your test methodology.

BTW - me doing the same test as you would not be peer review. Peer review means looking at your methodology and conclusions and providing comments and proposals for improvements (where applicable) - i.e. similar to what I tried to do up to now. :)
What you suggest I do is independently replicate your test and confirm / refute the validity of your results. However for people to be interested in replication you should first be able to show that your method is valid.

Anyway I think there's not much more I can help with so I'll drop out now. I do hope you found some of the information helpful and wish you good luck with your investigations!
As you can see in my post prior to your reply, I already updated my sentence before you post your last reply.

I also felt that using "arguing" is a bad choice of word. You have been sincerely offering suggestions and "arguing" is not the right word of choice.

I changed from "arguing" to "Instead of keep asking me doing the kind of blind tests that I can't do anytime soon".

Again, I truely appreciate you spending time giving suggestions. Without your suggestions, I wouldn't have motivated to do some frequency scans. This lead me to finding interesting high frequency slope from the Topping combo.

Please don't be upset. My fault entirely for using the wrong word.

Have a great day!
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Here is the freq response as captured with Focusrite Scarlett

The chain:
AMP jack -> jack to RCA cable -> RCA to XLR cable -> XLR input of Forte Scarlett

View attachment 190916

The different in db seems to match what I measured with multimeter at 90hz. It would seems I still can't find explanation why Topping combo gives me extra hit and Gustard combo is not.
Haha, regarding the Topping combo slope, I just realized it is normal because my E30 filter is somehow set to #2. I will check again with filter 3 when have time.

Johnyoung of Topping gave me the filters chart before at
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
I'm stepping in on the middle of this, but are these files all zeroes and then a click? Many DACs mute on zeroes, and vary how quickly they unmute. What you can do is mix clicks with constant white noise at -120 db. This will prevent muting while you'll not hear noise at that level.
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
I'm stepping in on the middle of this, but are these files all zeroes and then a click? Many DACs mute on zeroes, and vary how quickly they unmute. What you can do is mix clicks with constant white noise at -120 db. This will prevent muting while you'll not hear noise at that level.
Interesting!

I checked the web page source yesterday and could not find a way to download the test files. If you can find a way, please share your method.

The 5ms test link is at https://www.audiocheck.net/blindtests_timing_2w.php?time=5

Thanks!
 
OP
Pdxwayne

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
I'm stepping in on the middle of this, but are these files all zeroes and then a click? Many DACs mute on zeroes, and vary how quickly they unmute. What you can do is mix clicks with constant white noise at -120 db. This will prevent muting while you'll not hear noise at that level.
Hmm, thinking more about it after looking at my chart again at

Since Gustard x16 have the extra click clue with two other amps, I think it might not related to the mute on zero.

Thanks for giving ideas! I still would love to know how to download the test files.
: )
 
Top Bottom