Pdxwayne
Major Contributor
- Joined
- Sep 15, 2020
- Messages
- 3,219
- Likes
- 1,172
Did you even read post #196? I can't help you anymore. Feel free to ignore me.I think you need to read that thread more carefully.
Did you even read post #196? I can't help you anymore. Feel free to ignore me.I think you need to read that thread more carefully.
What does Prince Charles have to do with anything?Did you even read post #196? I can't help you anymore. Feel free to ignore me.
Haha, we are talking about a different thread. Don't just read the first post. On second try, OP and his friend could sense difference.What does Prince Charles have to do with anything?
I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.What does Prince Charles have to do with anything?
Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:2) That was done in a following manner: 10 swiches between different sources per song, subject knows that each time its an different source. Subject needs to determine which source is which.
Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch.
I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.
I hadnt been keeping close tabs on that thread but I have just added a point there that I dont think was picked up on https://www.audiosciencereview.com/...inguish-between-10-and-15k.31105/post-1108216I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.
@Pdxwayne IMHO the members here are giving valid criticism about the methodology as described in that post. Level matching with hand-held scope may or may not be precise enough, but the listening test methodology was problematic:
Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:
Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:
Still a very nice and praise-worthy exercise, IMHO!
I think you need to read that thread more carefully.
what about post 203?Did you even read post #196? I can't help you anymore. Feel free to ignore me.
Indeed, I have some doubts and asked more questions. That is why we have post #196 in that thread.I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.
@Pdxwayne IMHO the members here are giving valid criticism about the methodology as described in that post. Level matching with hand-held scope may or may not be precise enough, but the listening test methodology was problematic:
Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:
Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:
Still a very nice and praise-worthy exercise, IMHO!
My understanding is that, post #203 was about a tube amp he wanted to check later on. Unrelated to the listening chains he used for blind tests.what about post 203?
Also test was not ABX. Tests like this are fine if you just want to explore for yourself, no use for convincing other people. It's not trivial to set up a fully controlled test as I'm sure you're aware. You can't use this test as it stands as any sort of evidence to settle the wider question that there is really a difference.
Not saying that in a test with proper controls he would not be able to distinguish, but so far he's not quite managed that. I'd be happy to see him do that and get conclusive results, one way or another. My interest in it is entirely academic - it doesn't matter to me what the outcome is, only that it is a genuine outcome.
Just note that, while definitely a nice and interesting exercise (and surely helpful for the poster who did the test!), due to incomplete controls and lack of statistical significance it should IMHO not be considered evidence of audibility in the scientific sense, nor used to argue audibility in the wider discussion.Like you said, very praise-worthy exercise already.
To be honest, if the poster is happy, I see no reason to push them for more strict controls. If they ask for help with setup I'm sure many here would be happy to help (me included - though I'm sure there are far more competent members on this forum!)Feel free to continue suggest more in that thread and see if the OP willing to do even better controlled ABX.
Maybe so, that's not entirely clear. He does seem to be clear that the sources were different. Sorry, but I just have too many issues with the way it was done to agree that the results are conclusive.My understanding is that, post #203 was about a tube amp he wanted to check later on. Unrelated to the listening chains he used for blind tests.
Again, feel free to ask the OP about the test chains he used to get his results.
Hmm, why don't you ask more questionsJust note that, while definitely a nice and interesting exercise (and surely helpful for the poster who did the test!), due to incomplete controls and lack of statistical significance it should IMHO not be considered evidence of audibility in the scientific sense, nor used to argue audibility in the wider discussion.
...
Source to me, in that context, means there are two listening chains. Please, ask clarifying questions in his thread.Maybe so, that's not entirely clear. He does seem to be clear that the sources were different. Sorry, but I just have too many issues with the way it was done to agree that the results are conclusive.
Well, if somebody has a pair of massive golden ears, it must be Prince CharlesHaha, we are talking about a different thread.
But you realize that this is not really 10/10, right?If OP can do 10/10 each for 4 different songs, I would say it is highly significant.
Because I'm answering to your posts from this thread, OP didn't raise these questions (as far as I know, at least) Hope you still find it helpful.Hmm, why don't you ask more questions
to OP in his thread instead of arguing about statistically significant here?
Let's assume what you assumed was correct. He and his friend still did 7/8. Still not bad for for probability of 0.035.But you realize that this is not really 10/10, right?
According to the description from the author, there were 10 switches, and each time a switch was done the other DAC was playing (and they were also controlling the switching, so they knew when the change happened). So in principle if you get the first one right, you get all of them right - and if you get it wrong you get all of them wrong.
So therefore the 10 switches are really like a single trial, and every song is a new trial - so in principle it is 4/4, not 4 times 10/10 as you seem to suggest.
It is of course very different if the participants don't know after each switch whether the same or different DAC is playing - that would make each switch a trial
Because I'm answering to your posts from this thread, OP didn't raise these questions (as far as I know, at least) Hope you still find it helpful.
Unless I'm terribly misunderstanding something, it seems to be what the author describes:Let's assume what you assumed was correct.
2) That was done in a following manner: 10 swiches between different sources per song, subject knows that each time its an different source. Subject needs to determine which source is which.
8 trials is really not a lot (and note that one person got 4/4 the other 3/4); but even if we make the jump and assume significance - using different sources and things like lack of time sync or level differences could theoretically be easy 'tells' to help with source identification.He and his friend still did 7/8. Still not bad for for probability of 0.035.
As I said - I found the thread interesting as it is. If the poster wants to do more strict testing, great for them - it is a great way to learn!You can always ask them to repeat again to get to 14/16 if you wish.
; )
I agree that more can be done to eliminate tells.Unless I'm terribly misunderstanding something, it seems to be what the author describes:
8 trials is really not a lot (and note that one person got 4/4 the other 3/4); but even if we make the jump and assume significance - using different sources and things like lack of time sync or level differences could theoretically be easy 'tells' to help with source identification.
I find this article by @SIY gives a nice overview of the various things that one need to think about when preparing a blind test (including various inadvertent cues or 'tells'). E.g. see chapter 3. and especially subchapter 3e.
As I said - I found the thread interesting as it is. If the poster wants to do more strict testing, great for them - it is a great way to learn!
All I'm saying is we should be careful how we interpret the results of such informal tests (and especially if we wish to generalize them) - that of course doesn't invalidate the effort nor the value for the person performing the test!
The first thing that comes to my mind is making sure to all stacks are playing at the exact same level before running the test and see if you still find them different (e.g. measure the output with multimeter or ADC and match as closely as you can).I agree that more can be done to eliminate tells.
Talking about tells, I am pretty sure you saw my thread here:
DAC and amp combos did not give same clues when running online blind tests. Why? What would be the desired clue?
When running online blind timing tests yesterday, I heard different clues when using multiple combination of dac and amps. I wonder why. I thought, as long as I am using transparent DAC and amp, I can mix and match them and any combo will sound the same. But not this case. Is it okay? More...www.audiosciencereview.com
Any comments?
Thanks!The first thing that comes to my mind is making sure to all stacks are playing at the exact same level before running the test and see if you still find them different (e.g. measure the output with multimeter or ADC and match as closely as you can).
Second is of course not knowing which stack is playing when doing the test (needs a helper!), do notes for each (now anonymous) stack and then compare the results at the end.
Lastly, it is worth checking if all headphone amps you use have sufficiently low output impedance - if not that can significantly modulate the frequency response due to headphone impedance loading, and could be different between different headphone amps (see this post I did a while ago with some examples of the effect).
Hope this helps!
Well I guess if you're happy with the results you got then all is well. However if you want to try and understand the cause for these results, or at least want to see if they are repeatable under controlled conditions, then it makes sense to try to limit the number of variables and also to try and eliminate any source of potential bias.Thanks!
In this case, it doesn't make sense to use a constant level. It is about passing the online timing tests. I checked with all sort of volume levels to see if I can sense a clue in order to pass the tests. Certain combos are much easier than others to notice a difference, even when comparing with multiple volume levels.
To an extent yes, indeed - we need to take care about certain technical aspects when putting various audio devices in the same system to get good performance.Anyway, what I am trying to say is knowing a DAC is transparent is not enough. We can't hear a DAC directly without amp and headphones/speakers. The whole chain, DAC, amp and headphones/speakers combinations can make a difference, as you already notice.