Topping E30 DAC Review

[samsa]

I think the easiest is let both dac playing and use an ab switch box.

That would be fine for the "AB" part of the test (assuming we have the software on the PC to simultaneously play out of both USB ports), but how do you carry out the "X" part?

You keep acting as if this is completely trivial. Please show your work.

(To be completely honest, I don't see how an AB switch box is any improvement over switching RCA cables, but you're the one designing the protocol, so I'll indulge you.)

Edit: "[A]ssuming we have the software on the PC to simultaneously play out of both USB ports" is a nontrivial requirement. If this really is a requirement of your protocol, I'm going to have to ask for some examples of software that will allow you to do that.

 

[BDWoody]

Pain or no pain, could you sketch a protocol for how one would actually do a double blind ABX test of a pair of DACs in a home environment?

The same way you would do it in any other environment, which could include adding new equipment as needed. I've never gone so far as to do what would be considered a scientifically valid test, but I would if I thought I kept hearing differences before claiming I did, because...Science Forum...

With what I have in front of me here, I'd set up my DAC's with syc'd CCA's (optical obviously...unless I was comparing the CCA's analog out) out of roon, match the outputs by volt meter to within .1%, mix up the rca's so I don't know which is which, connect them to two inputs on preamp (Schiit Freya), and switch between inputs as I sit and listen. If I heard a difference I could reliably identify, I'd enlist some help and ask here for how to tighten it up for possible submission for the 'peer review' process that inevitably accompanies an unexpected, previously unidentified phenomenon being discovered as repeatable and verifiable.

Oh...Also needed is the ability to switch between DAC's in <1/4 second or so. So, stopping and switching wires won't work.

There isn't one answer, but without a very high level of controls, the results will be (considered) worthless. Bias is an insidious beast...

 

[samsa]

If I heard a difference I could reliably identify, I'd enlist some help and ask here for how to tighten it up ...

I'm asking here, and I'm not getting any response. How do you set up the "X" part of a double blind ABX test of a pair of DACs?

 

[BDWoody]

I'm asking here, and I'm not getting any response. How do you set up the "X" part of a double blind ABX test of a pair of DACs?

I'm honestly not trying to be evasive, as it isn't an easy one. Maybe 2 remote controls, with the switcher responding to a random number generator indicating one or the other choice on the "X" remote, which is also blind as to which is which...if that makes sense...

So, choice 'A', choice 'B', then choice '1' or '2' from the other remote, as determined by reading the RNG results. Blind to listener, and unknown to switcher.
Maybe something along that line, but I'd have to think through it...

Would be nice if there was an easier way to casually do it...open to suggestions.

 

[yejun]

I'm honestly not trying to be evasive, as it isn't an easy one. Maybe 2 remote controls, with the switcher responding to a random number generator indicating one or the other choice on the "X" remote, which is also blind as to which is which...if that makes sense...

So, choice 'A', choice 'B', then choice '1' or '2' from the other remote, as determined by reading the RNG results. Blind to listener, and unknown to switcher.
Maybe something along that line, but I'd have to think through it...

Would be nice if there was an easier way to casually do it...open to suggestions.

I think switcher just need to do the X switch based random number. I can't think of any reason why ABA or ABB have different effect on the switcher.

 

[BDWoody]

I think switcher just need to do the X switch based random number. I can't think of any reason why ABA or ABB have different effect on the switcher.

He knows whether the result was correct or not...

Edit: Not necessarily. Good point... I'm certainly not a testing expert .

 

[samsa]

I think switcher just need to do the X switch based random number. I can't think of any reason why ABA or ABB have different effect on the switcher.

If the switcher knows whether "X" is A or B, then that's single-blind, not double-blind. Even I could set up a single-blind ABX test.

@BDWoody was rather insistent that only a double-blind ABX test would suffice. So I'd like to know how one would set up such a test.

 

[yejun]

If the switcher knows whether "X" is A or B, then that's single-blind, not double-blind. Even I could set up a single-blind ABX test.

@BDWoody was rather insistent that only a double-blind ABX test would suffice. So I'd like to know how one would set up such a test.

The switcher doesn't know which equipment A or B connected to. I don't see any difference it will make.

 

[samsa]

The switcher doesn't know which equipment A or B connected to. I don't see any difference it will make.

It doesn't freaking matter what "A" and "B" are.

All that matters in an ABX test is whether the test subject can identify whether "X" is "A" or "B". If the switcher knows the answer, then the test isn't double-blind. And that means that it would be considered worthless hereabouts.

 

[yejun]

It doesn't freaking matter what "A" and "B" are.

All that matters in an ABX test is whether the test subject can identify whether "X" is "A" or "B". If the switcher knows the answer, then the test isn't double-blind. And that means that it would be considered worthless hereabouts.

If A or B has no special meaning to the switcher, both answer have the same meaning to the switcher before the result is revealed.

 

[BDWoody]

If the switcher knows whether "X" is A or B, then that's single-blind, not double-blind. Even I could set up a single-blind ABX test.

@BDWoody was rather insistent that only a double-blind ABX test would suffice. So I'd like to know how one would set up such a test.

Seems to be that you are trying to make it sound like an unreasonable standard because you don't know how you'd do one, or because I can't tell you exactly how you'd do one with whatever equipment you have, could buy, or even knowing what you are trying specifically to test. It would depend. It's possible, but it's a pain.

The reason I would 'insist' on it, before putting real money on it, is because that is the currently accepted standard of proof. Otherwise, it's just varying degrees of 'because I say so.'

Not sure why the attitude. If you are seriously looking at setting up a test, I would suggest posting a new thread with exactly what you are trying to test, and you will get feedback a lot more useful than what you'll find on page 112 of this thread.

 

[samsa]

Seems to be that you are trying to make it sound like an unreasonable standard because you don't know how you'd do one, or because I can't tell you exactly how you'd do one with whatever equipment you have, could buy, ...

I do think it's an unreasonable standard to ask others to adhere to, given that you can't tell them how to do so.

"Science" doesn't require unachievable conditions to make progress. We learn to live with the limitations of the experiments we can do rather than insisting that the only acceptable evidence comes from experiments that we can't do. I had thought that there was some straightforward protocol, that I was simply unaware of, for doing such an experiment. Evidently, I was wrong.

I would suggest that the scientific approach would be to either come up with such a protocol, or to accept that single-blind ABX is the best that can be done under the circumstances and be willing to settle for the evidence that it provides.

... or even knowing what you are trying specifically to test.

I thought we were clear on that: we were discussing blind ABX testing a pair of DACs.

 

[odyo]

Yet, you report results you don't believe you could duplicate without peeking...

Even with everything being 'super fresh,' if you could point out those differences with controls, you would be the first.

Blind tests are not perfect. Even if there is a difference you might not be able to spot it with the blind test. People shouldn't blindly trust those either. Blind tests might work with people who have more experience and training. It's really dreary process. Hearing is complicated.

 

[BDWoody]

I do think it's an unreasonable standard to ask others to adhere to, given that you can't tell them how to do so.

I disagree. If you aren't willing to do what is required to meet an accepted standard, the demand shouldn't be to lower the standard to a level that makes the results meaningless, it should be to figure out how to meet it. If you're actually serious, and not trying to be obtuse that is.

I would suggest that the scientific approach would be to either come up with such a protocol...,

I would say there's no 'either' if you want to demonstrate something to a degree that makes the result more than simply 'because I really really hear it,' and moves it into scientifically valid territory. Build a test. I believe I suggested one that might work. Whining that it's too hard doesn't change the science standard.

I thought we were clear on that: we were discussing blind ABX testing a pair of DACs.

What inputs? What kind of outputs? RCA/XLR or mixed? Are you comparing a range of sample rates? Just the easiest means of getting an output?
What other audio equipment do you have on hand? Are any transparent enough to act as a switching device for the DAC's in question? Etc...

Not one answer. Requires an understanding of why it is necessary to be worth it as a project.

 

[BDWoody]

Even if there is a difference you might not be able to spot it with the blind test.

Then, effectively, it isn't there, is it? If you claim you hear a difference, but can't under controlled testing, why should the conclusion be anything other than you aren't able to hear a difference. If the differences are all below an audible level, it doesn't matter.

 

[samsa]

I disagree. If you aren't willing to do what is required to meet an accepted standard, the demand shouldn't be to lower the standard to a level that makes the results meaningless, it should be to figure out how to meet it. If you're actually serious, and not trying to be obtuse that is.

I am serious.

What's unserious is demanding something with no idea of how to achieve it.

What inputs?

Since you mentioned Roon, let's say "Roon."

What kind of outputs? RCA/XLR or mixed?

Since this is a thread about the E30, let's say RCA.

Are you comparing a range of sample rates? Just the easiest means of getting an output?

I'm not sure how any of these questions are bringing you closer to suggesting a protocol for carrying out the test in question.

Clearly, if you suggested something that would work for one sample rate, it would also work for a range of sample rates.

What other audio equipment do you have on hand?

What other audio equipment would I need?

Are any transparent enough to act as a switching device for the DAC's in question? Etc...

Assume that whatever I have (or need to buy) are sufficiently transparent. If they're not, then the test will yield a null result.

None of these questions (except, perhaps, the one about cables) has the slightest bearing on answering how to carry out the requisite double-blind test.

 

[BDWoody]

I'm not sure how any of these questions are bringing you closer to suggesting a protocol for carrying out the test in question.
....
None of these questions (except, perhaps, the one about cables) has the slightest bearing on answering how to carry out the requisite double-blind test.

I'm really not sure how to help you. You seem to be quite riled up about expecting basic proof of claims, and think it's my job to develop your protocol. Are you saying it isn't possible to do it? Are you saying you can't, no matter how hard you try to ponder it, design a test that would meet the standard? It is just..impossible?

Or...you can't easily do it with what you've got sitting around? Or, it isn't obvious, and would require more time and money than you are willing to invest in finding the answer? If you believe it's impossible, that's it's own problem. If it's more hassle than you are willing to undertake, that a valid answer as well. Making your test protocol development my problem doesn't change the basic issue of the validity of poorly controlled subjective testing. A lower standard makes the results less than valid.

Rather than continue to take this further OT on this thread, I will exit here, and encourage you to start a new thread if you are interested in anything more than being mentally lazy and implying it is my problem to design a protocol for you, almost as if one not being readily apparent for your case, that somehow challenges the validity of a an accepted standard of proof.

I bet if someone said I'll give you $20,000 to design a testing protocol that would meet the standard you'd think about it for longer than it takes to type out another answer about how hard it is.
Maybe figure it out, or look into how others have by starting a new thread.

The claims of the differences heard between competent DAC's is not remotely suggested by any differences in measurements taken. If someone believes the difference is there, they should design a test to prove it. If not, thats not my problem. My standard is the scientific one, not the one limited by your resources to prove that you can hear things I don't think you can hear.

 

[Tup3x]

Then, effectively, it isn't there, is it? If you claim you hear a difference, but can't under controlled testing, why should the conclusion be anything other than you aren't able to hear a difference. If the differences are all below an audible level, it doesn't matter.

Not quite. It only tells that they sound close enough that the person has hard time noticing any difference. Now, we don't know how well that person remembers sounds or wether or not he/she is tone deaf (or for some reason he/she just doesn't do well in test like this). There could be still be clear difference and he/she might still be able to notice the difference with slightly different test arrangement.

The problem with audio is that you can't compare it like image. You could just put them side by side and see if there's a visual difference. It would be much easier to spot the differences. If you couldn't see them side by side it would be harder since you'd have to remember the previous image. It varies from person to person how well he/she remembers previous images or audio.

For that reason I'd say that A/B test would only tell if things are somewhat similar enough to not notice major difference. However, it doesn't mean that there is no difference. But if you can reliably tell the difference in controlled A/B test, then there really is clearly hearable difference (for you at least).

Basically audio A/B test might end up being just a audio memory test and not actual "can you spot any difference between x and y".

This is rather complicated subject in my opinion and besides the point of this thread. Just wanted to say what I thought about A/B testing.

 

[BDWoody]

Not quite. It only tells that they sound close enough that the person has hard time noticing any difference

I don't disagree fundamentally. But when we are hearing claims of some obvious and clearly apparent difference...like most seem to indicate they hear, isn't that going to be above the 'hard to notice a difference' realm? Depending on the testee, it could be a more coarse or fine tool, but I'd say it has to start with a serious effort at controlling for bias/placebo.

 

[odyo]

Then, effectively, it isn't there, is it? If you claim you hear a difference, but can't under controlled testing, why should the conclusion be anything other than you aren't able to hear a difference. If the differences are all below an audible level, it doesn't matter.

I meant audible difference.