Could someone help me to think through my ABX result using Bayesian reasoning?

[manisandher]

... test 1 and 2 did not seem to justify the thinking that the differences were 'clearly audible'.

So basically what you're saying is that tests 1 and 2 did not justify that the differences were 'clearly audible'... and neither did test 3.

OK, got it.

 

[solderdude]

You like to twist words in your favor it seems.

test 1 and 2 did not justify stating that the differences are clearly audible which makes test 3 the odd one out.
There are too little attempts in test 3 to justify assuming you clearly did hear differences, regardless of which theory you throw at it.
There are enough attempts in test one and 2 to justify saying you perceived something different than the harsh reality.

I also said that IF you really want to find out you will have to redo test 3 with statistically sufficient attempts. But I am not the first one stating that.

When you feel comfortable with getting 8 out of 8 correct in a row from a 30 attempt test and like validation of this by some reasoning it is fine by me. I thought you mentioned it bugged you. You said you wanted to find out. It seems like you don't want to really as you would have done more tests. Thinking you can measure it is a deception. When that could be done many snake-oil manufacturers would have ceased the opportunity to prove those close minded cloth eared engineers wrong already. Have not seen this happening.

 

[dshreter]

9/10 ABX result speaks for itself, it’s very unlikely but possible to happen by chance.

The community here cannot validate for you if the two samples were bit identical or if they were played back in a way that should otherwise be indistinguishable besides the obscure setting.

There is not much more to say. So, if you are confident in the test setup you can feel as though you hear the differences.

But what’s the point? Are you trying to make a decision about how to set something up? Does this relate to a purchase? Is there a golden USB cable that you want to validate actually sounds better?

On a more personal level if I had taken a train ride to help assist in ABX testing I would have been pretty upset if the person I came to help quit after a single 20 minute test pass unwilling to repeat the results. There’s a whole other side to this story, and when it comes to priors it has a very heavy influence on how I’m interpreting all of the data too.

 

[manisandher]

But ABX basically is ABA or ABB in which case you hear a difference between A and B and again from B to A.
In ABB case you hear a difference and going to B and then no difference.

Let's see if switching between A and B (even though I wasn't consciously aware of doing this) helps.

One point for every correct switch in tests 1 and 2 (irrespective of whether the response itself was actually correct):



That's 12/20. Still not great.

If we add the scores of the 3 tests together, that's (12+9)/30. 21/30. p=0.0133.

This is no different than XXXXX where ABB or ABA are followed. This was not consistent in the first 2 attempts so I see no reason to believe ABXABX is so much better than ABXXXXXXX.

You don't believe that ABXABXABXABXABXABXABXABXABXABX is better than ABXXXXXXXXXX? Sorry if I'm misunderstanding.

 

[manisandher]

Thinking you can measure it is a deception. When that could be done many snake-oil manufacturers would have ceased the opportunity to prove those close minded cloth eared engineers wrong already. Have not seen this happening.

Well, I've got an idea. It may well not work. But if it does, I'll explain what I did, and it'll be fully replicable by anyone else.

 

[manisandher]

9/10 ABX result speaks for itself, it’s very unlikely but possible to happen by chance.

It's a 1 in a 100 chance. There's no ambiguity there at all.

On a more personal level if I had taken a train ride to help assist in ABX testing I would have been pretty upset if the person I came to help quit after a single 20 minute test pass unwilling to repeat the results.

I didn't find concentrating for 54x 15 second samples (with ~20 seconds between each as the software settings were changed) very easy at all.

After the 3 tests, we had lunch. We took some measurements from the analogue output of the DAC for post analysis. I sat Mans down for a listen to talk him through what I was hearing between A and B.

I think we were both happy to have wrapped up when we did.

There was zero animosity between us when Mans left...

 

[mansr]

9/10 ABX result speaks for itself, it’s very unlikely but possible to happen by chance.

It's still the most likely explanation anyone has offered.

 

[solderdude]

You don't believe that ABXABXABXABXABXABXABXABXABXABX is better than ABXXXXXXXXXX? Sorry if I'm misunderstanding.

No need to be sorry... when there are differences that can be heard why would ABXXXXXX be invalid ?
I mean X1 and X2 have a difference or they don't = detectable and so forth for X2, X3 etc.
It is about differences not preferences nor correct assignment.
One can be 100% wrong in assignment yet 100% correct in hearing differences.

I believe in blind testing though. Properly conducted...

 

[solderdude]

Well, I've got an idea. It may well not work. But if it does, I'll explain what I did, and it'll be fully replicable by anyone else.

Do you mind explaining your experiment idea. Some people might shoot holes in it which could save you the trouble. Instead you could try listening controlled with your ABXABX method with statistical relevance and witnessed.
It's about your gears and your ears. Measurement equipment can't prove any of this.

 

[dshreter]

Exactly. I said it’s unlikely. So…. now what? You heard a difference between two stimuli.

Is there a setting you want us to know can better replicate music? Want someone to say you’re right and Mansr is wrong? Validate that you are an elite listener? Get to the point

 

[manisandher]

Get to the point

WTF?

 

[dshreter]

WTF?

Yeah, let’s suppose your hypothesis is supported and you can hear the difference in these two stimuli. So what?

Move the conversation ahead.

 

[danadam]

Get to the point

I think the point was the 2nd one:

2. I wanted to understand what could be causing what I was hearing, with the hope that an interesting insight (a way of further improving the SQ of digital audio perhaps?) might be gained

 

[dshreter]

I think the point was the 2nd one:

Ok. Have A and B been clearly specified? Let’s discuss what they are and why they might or might not sound different.

Statistical inference works. There’s nothing more to say about that unless OP wants to conduct more trials or move on and accept the hypothesis. Arguing in circles about the test is pointless. He got a low probability outcome and it just has to be taken on faith unless the test conducted in the past is to be inspected.

 

[solderdude]

The problem seems to be that the cause is still not determined which hangs on audibility that is still not conclusive.

The latter can only be done by Mani as others are unable to hear it (or are silent as to not stir the pot and get the comments Mani has to endure).

So... IMO the only way forward is to redo the test (it is 3 years later now) with agreed upon conditions and get it moving again instead of trying to justify a 10 attempt test that might be a fluke given the earlier attempts.
All doubt must be removed first. It isn't regardless of the theories let loose on the issue.

For now it is more like a magic trick that hasn't been explained yet. Some believe it is impossible, others see it as clear evidence.

 

[voodooless]

Ok. Have A and B been clearly specified? Let’s discuss what they are and why they might or might not sound different.

They were never clearly specified. Apparently it’s a difference in the SFS setting in the audio player XXHighEnd. SFS seems to stand for Split File Size. All kinds of wacky shit can be found on this. None of it makes any sense whatsoever.

 

[manisandher]

Yeah, let’s suppose your hypothesis is supported and you can hear the difference in these two stimuli. So what?

Like I said from the beginning, I have nothing to sell or to promote.

But if my hypothesis is supported (in this case, that a bit-identical change in a software player can cause an audible difference, with everything else remaining constant), I would have thought that this would be of very great interest here on ASR.

It would mean that the current measurements (SINAD, IMD, etc.) taken to compare two streamers, lets say, although certainly not incorrect, may be inadequate.

For my own part, I would love to find measurements that correlate with what I hear, so that I might further optimise my system.

 

[manisandher]

They were never clearly specified. Apparently it’s a difference in the SFS setting in the audio player XXHighEnd. SFS seems to stand for Split File Size. All kinds of wacky shit can be found on this. None of it makes any sense whatsoever.

Look, we could have chosen another variable, e.g., USB cables. But I went for a variable the effect of which I was very familiar. Any of the 100s of regular XXHighEnd users would instantly recognise my description of the piano attack sounding 'soft' with a larger SFS, and 'sharp' with a low SFS.

And yet the SFS setting does not change the bits in any way, shape or form.

 

[voodooless]

For my own part, I would love to find measurements that correlate with what I hear, so that I might further optimise my system.

Let’s not turn this on it’s head. Let’s first without a shadow of doubt agree that you actually can hear a difference. If so, we can figure out why.

 

[Raindog123]

I suggest @manisandher to truncate his third (9/10) run to just 8 tries, and claim it “whopping 100%“ (8/8)!

Not to mention that even by his own logic - by counting the very first “X” attempt (after the initial “AB”) in the first two series - his results and the data set will be “10/12“ (an incorrect first guess in the first series and a correct one in the second). So - 10 out of 12 - is not that convincing after all.

But who cares, with his “indisputably massive” prior!?