Serious Question: How can DAC's have a SOUND SIGNATURE if they measure as transparent? Are that many confused?

[Pdxwayne]

I think you need to read that thread more carefully.

Did you even read post #196? I can't help you anymore. Feel free to ignore me.

 

[voodooless]

Did you even read post #196? I can't help you anymore. Feel free to ignore me.

What does Prince Charles have to do with anything?

 

[Pdxwayne]

What does Prince Charles have to do with anything?

Haha, we are talking about a different thread. Don't just read the first post. On second try, OP and his friend could sense difference.

This is link to #196 to that thread:

DAC ABX shootout - unable to distinguish between 10$ and 15k$

I know its probably me, but I cannot fathom an apple dongle sounding the same as any Chord or Topping DAC. Yet... it does. The performance of that little thing is astonishingly good. Not state of the art measurement, but damn excellent measurements. When I did a comparison to an RME ADI-2 Pro...

www.audiosciencereview.com

 

[dominikz]

What does Prince Charles have to do with anything?

I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.

@Pdxwayne IMHO the members here are giving valid criticism about the methodology as described in that post. Level matching with hand-held scope may or may not be precise enough, but the listening test methodology was problematic:

2) That was done in a following manner: 10 swiches between different sources per song, subject knows that each time its an different source. Subject needs to determine which source is which.

Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:

The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch.

Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Still a very nice and praise-worthy exercise, IMHO!

 

[Jimbob54]

I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.

@Pdxwayne IMHO the members here are giving valid criticism about the methodology as described in that post. Level matching with hand-held scope may or may not be precise enough, but the listening test methodology was problematic:

Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:

Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:


Still a very nice and praise-worthy exercise, IMHO!

I hadnt been keeping close tabs on that thread but I have just added a point there that I dont think was picked up on https://www.audiosciencereview.com/...inguish-between-10-and-15k.31105/post-1108216

 

[Mart68]

I think you need to read that thread more carefully.

Did you even read post #196? I can't help you anymore. Feel free to ignore me.

what about post 203?

Also test was not ABX. Tests like this are fine if you just want to explore for yourself, no use for convincing other people. It's not trivial to set up a fully controlled test as I'm sure you're aware. You can't use this test as it stands as any sort of evidence to settle the wider question that there is really a difference.

Not saying that in a test with proper controls he would not be able to distinguish, but so far he's not quite managed that. I'd be happy to see him do that and get conclusive results, one way or another. My interest in it is entirely academic - it doesn't matter to me what the outcome is, only that it is a genuine outcome.

 

[Pdxwayne]

I do believe @Pdxwayne is talking about the other thread, specifically: post #196 in that thread.

@Pdxwayne IMHO the members here are giving valid criticism about the methodology as described in that post. Level matching with hand-held scope may or may not be precise enough, but the listening test methodology was problematic:

Since the listener knows that each time there is a switch (and switch time is known) a different source is playing, the 10 switches boil-down to a single test. So in reality for 4 songs it is 4 identifications in total, not 40. In addition the test was apparently not double-blind, if I understood the methodology correctly, and there may have been time-sync issues between the DACs:

Even the author expressed doubts if they would be able to perform as well in a double-blind ABX:


Still a very nice and praise-worthy exercise, IMHO!

Indeed, I have some doubts and asked more questions. That is why we have post #196 in that thread.
: )

Like you said, very praise-worthy exercise already.

Feel free to continue suggest more in that thread and see if the OP willing to do even better controlled ABX.

 

[Pdxwayne]

what about post 203?

Also test was not ABX. Tests like this are fine if you just want to explore for yourself, no use for convincing other people. It's not trivial to set up a fully controlled test as I'm sure you're aware. You can't use this test as it stands as any sort of evidence to settle the wider question that there is really a difference.

Not saying that in a test with proper controls he would not be able to distinguish, but so far he's not quite managed that. I'd be happy to see him do that and get conclusive results, one way or another. My interest in it is entirely academic - it doesn't matter to me what the outcome is, only that it is a genuine outcome.

My understanding is that, post #203 was about a tube amp he wanted to check later on. Unrelated to the listening chains he used for blind tests.

Again, feel free to ask the OP about the test chains he used to get his results.

 

[dominikz]

Like you said, very praise-worthy exercise already.

Just note that, while definitely a nice and interesting exercise (and surely helpful for the poster who did the test!), due to incomplete controls and lack of statistical significance it should IMHO not be considered evidence of audibility in the scientific sense, nor used to argue audibility in the wider discussion.

Feel free to continue suggest more in that thread and see if the OP willing to do even better controlled ABX.

To be honest, if the poster is happy, I see no reason to push them for more strict controls. If they ask for help with setup I'm sure many here would be happy to help (me included - though I'm sure there are far more competent members on this forum!)

 

[Mart68]

My understanding is that, post #203 was about a tube amp he wanted to check later on. Unrelated to the listening chains he used for blind tests.

Again, feel free to ask the OP about the test chains he used to get his results.

Maybe so, that's not entirely clear. He does seem to be clear that the sources were different. Sorry, but I just have too many issues with the way it was done to agree that the results are conclusive.

 

[Pdxwayne]

Just note that, while definitely a nice and interesting exercise (and surely helpful for the poster who did the test!), due to incomplete controls and lack of statistical significance it should IMHO not be considered evidence of audibility in the scientific sense, nor used to argue audibility in the wider discussion.

...

Hmm, why don't you ask more questions
to OP in his thread instead of arguing about statistically significant here?

If OP can do 10/10 each for 4 different songs, I would say it is highly significant.

 

[Pdxwayne]

Maybe so, that's not entirely clear. He does seem to be clear that the sources were different. Sorry, but I just have too many issues with the way it was done to agree that the results are conclusive.

Source to me, in that context, means there are two listening chains. Please, ask clarifying questions in his thread.

 

[voodooless]

Haha, we are talking about a different thread.

Well, if somebody has a pair of massive golden ears, it must be Prince Charles

 

[dominikz]

If OP can do 10/10 each for 4 different songs, I would say it is highly significant.

But you realize that this is not really 10/10, right?

According to the description from the author, there were 10 switches, and each time a switch was done the other DAC was playing (and they were also controlling the switching, so they knew when the change happened). So in principle if you get the first one right, you get all of them right - and if you get it wrong you get all of them wrong.
So therefore the 10 switches per song are really like a single trial, and every song is like a new trial - so in principle it is 4/4 with 4 songs, not 4 times 10/10 as you seem to suggest.

It is of course very different if the participants don't know after each switch whether the same or different DAC is playing - that would make each switch a trial

Hmm, why don't you ask more questions
to OP in his thread instead of arguing about statistically significant here?

Because I'm answering to your posts from this thread, OP didn't raise these questions (as far as I know, at least) Hope you still find it helpful.

 

[Pdxwayne]

But you realize that this is not really 10/10, right?

According to the description from the author, there were 10 switches, and each time a switch was done the other DAC was playing (and they were also controlling the switching, so they knew when the change happened). So in principle if you get the first one right, you get all of them right - and if you get it wrong you get all of them wrong.
So therefore the 10 switches are really like a single trial, and every song is a new trial - so in principle it is 4/4, not 4 times 10/10 as you seem to suggest.

It is of course very different if the participants don't know after each switch whether the same or different DAC is playing - that would make each switch a trial


Because I'm answering to your posts from this thread, OP didn't raise these questions (as far as I know, at least) Hope you still find it helpful.

Let's assume what you assumed was correct. He and his friend still did 7/8. Still not bad for for probability of 0.035.

You can always ask them to repeat again to get to 14/16 if you wish.
; )

 

[dominikz]

Let's assume what you assumed was correct.

Unless I'm terribly misunderstanding something, it seems to be what the author describes:

2) That was done in a following manner: 10 swiches between different sources per song, subject knows that each time its an different source. Subject needs to determine which source is which.

He and his friend still did 7/8. Still not bad for for probability of 0.035.

8 trials is really not a lot (and note that one person got 4/4 the other 3/4); but even if we make the jump and assume significance - using different sources and things like lack of time sync or level differences could theoretically be easy 'tells' to help with source identification.

I find this article by @SIY gives a nice overview of the various things that one need to think about when preparing a blind test (including various inadvertent cues or 'tells'). E.g. see chapter 3. and especially subchapter 3e.

You can always ask them to repeat again to get to 14/16 if you wish.
; )

As I said - I found the thread interesting as it is. If the poster wants to do more strict testing, great for them - it is a great way to learn!

All I'm saying is we should be careful how we interpret the results of such informal tests (and especially if we wish to generalize them) - that of course doesn't invalidate the effort nor the value for the person performing the test!

 

[Pdxwayne]

Unless I'm terribly misunderstanding something, it seems to be what the author describes:



8 trials is really not a lot (and note that one person got 4/4 the other 3/4); but even if we make the jump and assume significance - using different sources and things like lack of time sync or level differences could theoretically be easy 'tells' to help with source identification.

I find this article by @SIY gives a nice overview of the various things that one need to think about when preparing a blind test (including various inadvertent cues or 'tells'). E.g. see chapter 3. and especially subchapter 3e.


As I said - I found the thread interesting as it is. If the poster wants to do more strict testing, great for them - it is a great way to learn!

All I'm saying is we should be careful how we interpret the results of such informal tests (and especially if we wish to generalize them) - that of course doesn't invalidate the effort nor the value for the person performing the test!

I agree that more can be done to eliminate tells.

Talking about tells, I am pretty sure you saw my thread here:

DAC and amp combos did not give same clues when running online blind tests. Why? What would be the desired clue?

When running online blind timing tests yesterday, I heard different clues when using multiple combination of dac and amps. I wonder why. I thought, as long as I am using transparent DAC and amp, I can mix and match them and any combo will sound the same. But not this case. Is it okay? More...

www.audiosciencereview.com

Any comments?

 

[dominikz]

I agree that more can be done to eliminate tells.

Talking about tells, I am pretty sure you saw my thread here:

DAC and amp combos did not give same clues when running online blind tests. Why? What would be the desired clue?

When running online blind timing tests yesterday, I heard different clues when using multiple combination of dac and amps. I wonder why. I thought, as long as I am using transparent DAC and amp, I can mix and match them and any combo will sound the same. But not this case. Is it okay? More...

www.audiosciencereview.com

Any comments?

The first thing that comes to my mind is making sure to all stacks are playing at the exact same level before running the test and see if you still find them different (e.g. measure the output with multimeter or ADC and match as closely as you can).
Second is of course not knowing which stack is playing when doing the test (needs a helper!), do notes for each (now anonymous) stack and then compare the results at the end.
Lastly, it is worth checking if all headphone amps you use have sufficiently low output impedance - if not that can significantly modulate the frequency response due to headphone impedance loading, and could be different between different headphone amps (see this post I did a while ago with some examples of the effect).

Hope this helps!

 

[Pdxwayne]

The first thing that comes to my mind is making sure to all stacks are playing at the exact same level before running the test and see if you still find them different (e.g. measure the output with multimeter or ADC and match as closely as you can).
Second is of course not knowing which stack is playing when doing the test (needs a helper!), do notes for each (now anonymous) stack and then compare the results at the end.
Lastly, it is worth checking if all headphone amps you use have sufficiently low output impedance - if not that can significantly modulate the frequency response due to headphone impedance loading, and could be different between different headphone amps (see this post I did a while ago with some examples of the effect).

Hope this helps!

Thanks!

In this case, it doesn't make sense to use a constant level. It is about passing the online timing tests. I checked with all sort of volume levels to see if I can sense a clue in order to pass the tests. Certain combos are much easier than others to notice a difference, even when comparing with multiple volume levels.

Anyway, what I am trying to say is knowing a DAC is transparent is not enough. We can't hear a DAC directly without amp and headphones/speakers. The whole chain, DAC, amp and headphones/speakers combinations can make a difference, as you already notice.

 

[dominikz]

Thanks!

In this case, it doesn't make sense to use a constant level. It is about passing the online timing tests. I checked with all sort of volume levels to see if I can sense a clue in order to pass the tests. Certain combos are much easier than others to notice a difference, even when comparing with multiple volume levels.

Well I guess if you're happy with the results you got then all is well. However if you want to try and understand the cause for these results, or at least want to see if they are repeatable under controlled conditions, then it makes sense to try to limit the number of variables and also to try and eliminate any source of potential bias.
If you get consistent results no matter how much more difficult you try and make it for yourself, then you might be on to something

Anyway, what I am trying to say is knowing a DAC is transparent is not enough. We can't hear a DAC directly without amp and headphones/speakers. The whole chain, DAC, amp and headphones/speakers combinations can make a difference, as you already notice.

To an extent yes, indeed - we need to take care about certain technical aspects when putting various audio devices in the same system to get good performance.
E.g. we need to take care that the DAC output level matches input sensitivity of the headphone amp, and that the headphone amp output power is high enough, and output impedance low enough, for the headphones we wish to drive.
These days we have a lot of resources to help us estimate what might work well together - even before buying
And thankfully a lot of these devices also seem to be from the start designed in a way to make them quite universal (but not all are!).