• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Could someone help me to think through my ABX result using Bayesian reasoning?

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
Priors:
1. For many years, I'd been certain that I could hear clear differences between bit-identical playback means (e.g., different USB cables, streaming vs. local playback, different software players and their bit-identical settings, etc.) - something that has bugged me no end over the years
2. I wanted to understand what could be causing what I was hearing, with the hope that an interesting insight (a way of further improving the SQ of digital audio perhaps?) might be gained
3. I was confident that I could demonstrate what I was hearing in a controlled listening test
4. I invited @mansr to my place (paid for his train ticket, no less) to help conduct a listening test (I chose 2 different but bit-identical setttings in a software player as the single variable)
5. the test was almost double-blind
- Mans controlled the test from my office, two doors and a corridor away from me in my listening room (I could not see or hear anything he was doing)
- Mans used a random generator on his phone to determine whether X=A or X=B
6. I was expecting to get close to 100% correct, so clear were the differences I'd been hearing for years

Likelihood:
- I scored 9/10 in the only ABX we conducted
- I got #9 incorrect - we were 15 minutes into ABX at that point, and I was feeling fatigued (we'd conducted 2 non-ABX tests beforehand - see below)
- input into DAC during ABX was shown to remain bit-identical*

Commoness:
- p = 0.01

Disclosure:
- we'd conducted 2 non-ABX tests before the ABX, where I scored no better than guessing
- even here, I was certain I could hear differences, but had no reference against which to assess, so simply guessed A or B in my responses
- I don't think it's fair to include the results of these non-ABX tests in the ABX results (also, I'd never been involved in any tests beforehand, and was adjusting to the strange/unnatural setup)
- the results of these two non-ABX tests suggest there were no 'tells' during any of the tests
- * there was a glitch in the auto-start function of the digital recorder being used, so the first few ms (out of ~15,000 ms) of each digital capture had to be ignored
- there were no conflicts of interest on my part - I was neither selling nor promoting any product/service (I'm not a manufacturer or fan of any particular product)
- this test was conducted out of pure curiosity on my part
- I HATE that these differences seem to exist (to me, at least) - I do NOT want USB cables or different software players to sound different... and yet they seem to (to me)

It's not inconcievable that someone selected randomly from the street would score 9/10 in the first ABX they sat - there's a 1 in 100 chance that they would.

However... given the Priors above, surely the 9/10 result shouldn't be easily dismissed as a simple fluke?

Could someone help me think this through using Bayesian reasoning please?

FWIW, I'm still certain I can hear differences in bit-identical replay, to my genuine chagrin. No more ABX tests for me though - I hated the whole experience. However, using a low-noise DAC-ADC chain and @pkane 's DeltaWave, I'm pretty confident I can measure these differences now. I intend to do just that over the Christmas break, and will come back with what I find.

Mani.
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
Wow!

The fact that I was willing to pay for your train ticket is an important part of the Priors. Sorry if that's lost on you.

And in any event, I don't measure my wealth in terms of what I own, but in terms of the people in my life (hence the tattoo on my arm).

(I still have the 911_50 though :))

Mani.
 

MRC01

Major Contributor
Joined
Feb 5, 2019
Messages
2,433
Likes
2,660
Location
Pacific Northwest
Bayesian inference is a double-edged sword. The benefits are well known. Yet if the prior information is wrong, mis-interpreted, or mis-applied, it works against you. Choose your priors carefully. Most of the priors you mention can't be quantified anyway.

Why not take it at face value? 9 of 10 correct is 98.93% confidence. Even if the sources were bit identical, there must have been something different. Whatever it was, odds are, you heard it.

BTW, what exactly was different?
 

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,249
Likes
1,161
I think you should not consider practice runs.

What exactly are the chains? How do you switch between the chain? What kind of difference did you listen for?
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
Most of the priors you mention can't be quantified anyway.

How about my expectation of getting 100% correct (based on my experience over the years)? (Hence why I set up the test... hence why I paid for train ticket.)

BTW, what exactly was different?

The difference was a particular setting in the software player (XXHighEnd), known as 'SFS'. As far as I can tell (the developer is coy), different SFS settings 'load' the playback PC in different ways... but maintain the digital signal entering the DAC (as proven by the bit-identical captures from the digital recorder during the ABX).

The hypothesis is then that different levels of noise manage to make their way into the DAC due to the different PC loads (mechanism unknown), which have different effects on the digital-to-analogue conversion (mechanism unknown).

These effects don't seem to appear on any current measurements at the output of the DAC though, including jitter. My hypothesis here would be that the effects are amplified in highly varying signals with transients, like music, as opposed to steady-state signals like single- or multi-tone sine waves.

Mani.
 

MRC01

Major Contributor
Joined
Feb 5, 2019
Messages
2,433
Likes
2,660
Location
Pacific Northwest
The difference was a particular setting in the software player (XXHighEnd), known as 'SFS'. As far as I can tell (the developer is coy), different SFS settings 'load' the playback PC in different ways... but maintain the digital signal entering the DAC (as proven by the bit-identical captures from the digital recorder during the ABX). ...
You said "the difference" which implies no other differences - everything else identical?
The DAC was running on the PC? If so, the idea is that CPU/RAM loading of the PC changes its power consumption, which can cause slight variations in DC voltages which can affect a DAC's output?
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
From reading the first post I had assumed the two of you were friends, but you don’t sound like friends

We could have been, perhaps. But after the test, some animosity seemed to have crept in. After positing a number of other explanations for the 9/10 result, all of which were refuted by data, Mans simply claimed that it must have therefore been a fluke.

After all the trouble I'd been through getting the test set up and then actually done, I suppose I took some umbrage with this. And also, cheap 'n' cheerful as it was (conducted in my home - a domestic setting after all), I felt that he was dismissing something that was possibly valuable and insightful.

Mani.
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
You said "the difference" which implies no other differences - everything else identical?

Yes, everything else identical.

The DAC was running on the PC?

No. DAC had own power supply. In the case of the test, the DAC was connected to PC via BNC spdif.
 

MRC01

Major Contributor
Joined
Feb 5, 2019
Messages
2,433
Likes
2,660
Location
Pacific Northwest
Perhaps CPU/RAM loading of the PC caused timing issues creating jitter in the SPDIF data the PC sent to the DAC, and the DAC did not do a sufficient job re-clocking the data?

Without knowing details of the setup, this is mere speculation. If you're looking for opinions on where an audible difference could have emerged, it would be helpful to have a diagram of the setup.
 

Killingbeans

Major Contributor
Joined
Oct 23, 2018
Messages
2,286
Likes
3,780
Location
Bjerringbro, Denmark.
My hypothesis here would be that the effects are amplified in highly varying signals with transients, like music, as opposed to steady-state signals like single- or multi-tone sine waves.

In other words: A non-flat frequency response? That should be pretty easy to measure.

- I scored 9/10 in the only ABX we conducted

What was the reason you only did one?
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
If you're looking for opinions on where an audible difference could have emerged, it would be helpful to have a diagram of the setup.

No, actually I'm not. This test was conducted a few years ago now, and is pretty much done and dusted I think.

But I'd like to learn more about Bayesian reasoning, because I suspect it will apply when I do some measurements over Christmas. I just thought this test would be a useful example from me to learn from.
 

pkane

Major Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
3,190
Likes
5,642
Location
North-East
No. DAC had own power supply. In the case of the test, the DAC was connected to PC via BNC spdif.
The hypothesis is then that different levels of noise manage to make their way into the DAC due to the different PC loads (mechanism unknown), which have different effects on the digital-to-analogue conversion (mechanism unknown).

This didn't test for this specific hypothesis. Other differences, caused by some "unknown" software setting, could've resulted in increased jitter, for example. There were no tests done to demonstrate that PC noise was the cause.
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
What kind of difference did you listen for?
In other words: A non-flat frequency response? That should be pretty easy to measure.

The differences I (still) hear between various bit-identical replay schemes have nothing to do with frequency response. It's ususally the leading edges of instruments and voices that are the give away for me. In the case of the ABX, it was the leading edges of piano notes.
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
This didn't test for this specific hypothesis... There were no tests done to demonstrate that PC noise was the cause.

Agreed.

Other differences, caused by some "unknown" software setting, could've resulted in increased jitter, for example.

Any idea how a software setting might be able to cause an increase in jitter (keeping everything bit-identical)?
 

Killingbeans

Major Contributor
Joined
Oct 23, 2018
Messages
2,286
Likes
3,780
Location
Bjerringbro, Denmark.
The differences I (still) hear between various bit-identical replay schemes have nothing to do with frequency response.

You were talking about transients in music. They are a sum of single frequencies. A D-A conversion doesn't care about the "bustle" of music. If all of the frequency content is treated equally, then you have nothing to worry about.

It's ususally the leading edges of instruments and voices that are the give away for me. In the case of the ABX, it was the leading edges of piano notes.

Feel free to call me dumb, but what on earth is a "leading edge" of an instrument? o_O
 
OP
manisandher

manisandher

Senior Member
Joined
Nov 6, 2016
Messages
367
Likes
263
Location
Royal Leamington Spa, UK
Top Bottom