Double Blind Testing FAQ Development

[j_j]

I think if we can make a simple switch box for analog sources that costs less than $50, we can always lend them one. Wonder how much work it is to have a simple tone generator and a digital meter for the output.

Thing is, tone generator does not even have to be clean in order to be used to measure level. 3.5 digit AC digital voltmeters are remarkably cheap nowdays. The tone generator does not even have to be super-stable level, as long as there's no drift inside of 5 minutes. You just have to get the levels right and leave it at that.

Question is how does one randomize A/B, or alternately (and better) X? That requires some thinking, maybe.

Having a button you push to "set up test" that simply has a high frequency oscillator (of indeterminate frequency, really) pick X from one of A or B via flipflop (having A and B fixed and known is the best setup for a test) isn't truly random, but it's quite close. If you're going to use a box like this, you'll want to have break before make, unfortunately. It would be nice to be able to use analog switches, but they frankly aren't that good, so a dry relay contact is the better choice. This means you have "clicks" from signal interruption, which does lower the test sensitivity somewhat, but it's not catastrophic except for very, very sensitive tests. It's certainly better than most options. This doesn't address looping, but that's a different hard problem.

If you're doing this, then you only need moderate time alighment, a dozen samples here or there isn't a problem. 10 milliseconds would be, though, there are drummers who would notice in two beats, at least.

If you do it that way, make the time delay (for the no signal part) fixed if you can.

 

[amirm]

Question is how does one randomize A/B, or alternately (and better) X? That requires some thinking, maybe.

Someone built a hardware USB relay ABX switcher using Python running on the PC for control. Seems to be a great solution to me as it keeps the cost down. See: https://github.com/oxtoacart/pyabx

He put in some logic to click the relays like mad to keep the "tell" from being audible.

 

[restorer-john]

Break before make 4PDT relays for the speakers/balanced headphones etc. It doesn't matter about the relay noise- any selection has the same relay noise.

 

[j_j]

Break before make 4PDT relays for the speakers/balanced headphones etc. It doesn't matter about the relay noise- any selection has the same relay noise.

You always must "break" even if A=X and you switch between A and X. So you need two relays, one for A and one for B (to connect to X).

While there could also be loading issues, this isn't perfect, but it's not too bad.

 

[solderdude]

You can have 2 relays where one always becomes activated and another always is deactivated this should always make the same 'sound'.
When doing AA for instance you can simply click 1 relay shortly which should sound very similar.

Still such a device would only be interesting for those interested in truth finding and that use it properly. It cannot be used to prove someone else they do or do not have special hearing capabilities as it would be really easy to rig the test (intentionally or by accident).

 

[restorer-john]

You always must "break" even if A=X and you switch between A and X. So you need two relays, one for A and one for B (to connect to X).

I have five 4PDT relays in my comparator for 5 amplifier switching. I can route amp 1 or amp 2 to any single or multiples of them and switch between. There is always 2 relays clicking at once. The pole distance means they are always break before make. Not so good for the tube guys.

Not ABX of course- purely self cancelling pushbutton arrays driving the relays, but the principle is the same. The pushbutton arrays can easily be replaced with logic and output monitoring. It's really good fun to get someone to set up the amplifier inputs and not tell me. They could be all the same, all different and all the combinations in between.

Why limit yourself to just A, B and X? Relays are cheap and once the front end logic is sorted you could stack up multiple amps and compare a whole bunch of them in one go.[/QUOTE]

 

[j_j]

Still such a device would only be interesting for those interested in truth finding and that use it properly. It cannot be used to prove someone else they do or do not have special hearing capabilities as it would be really easy to rig the test (intentionally or by accident).

Oh, so true.

 

[restorer-john]

@j_j in a standalone ABX hardware comparator, are the original identities of A and B randomly set at the commencement of the test and maintained for the duration of the test?

 

[j_j]

@j_j in a standalone ABX hardware comparator, are the original identities of A and B randomly set at the commencement of the test and maintained for the duration of the test?

Typically, people will respond more accurately if the identity of A and B does not change in a set of trials.

HOWEVER, you should change it for the next set of trials (and tell the subject that) in order to make sure there is nothing funky going in.

 

[KSTR]

I have one of those lying around: https://www.sparkfun.com/products/retired/766
That, or something similar, would certainly suffice for a simple automated and logging ABX-Box. The software is least problematic part of such a project (at least for me, I hate mechanical DIYing).


Then again, copying a post of mine from the DBT volunteer thread, wrt DACs:

I would think the only way to have an invulnerable result here is building a microcontroller/RPi-based 5-button ABX-box with a sealed compartment containing the two DACs, level matched etc. Only three connections from this black box to the world: Mains input, Digital input (SPDIF coaxial or AES3 as that can be easily split), analog out (XLR or RCA) to headphone/speaker amp.
The advantage of this that the testing person has all freedoms, is not subject to "test stress" from being observed (either by a person or a webcam), etc.

Not exactly easy but not impossible in a joint forum effort. Details matter, for example any relays must be located in a sound-proofed section so they don't give clues. Hidden clues of various sorts are the main focus, this must be avoided at all cost.

There would some requirements for the DACs:
- must power up by applying mains (otherwise the controller would also need "to press the power/standby button" in some way).
- must have Digital Input other than USB. The test set's input could be USB though when an extra interface is used to distribute the USB signal to two DAI outputs simultaneously.

 

[Spkrdctr]

I have too, but I don't waste my time any more on idiots

That is the entire problem. Wild*ass claims are put forth by friendly idiots who don't know better. I too have given up on them.

 

[solderdude]

All one can do is kindly show them the error of their ways and try to convince them nicely and with respect how they could improve their test methods. When they don't want to.. uh-huh them and move on.

When they are eager to learn they can be shown methods that may work for them with minimal effort.

 

[Spkrdctr]

When they are eager to learn they can be shown methods that may work for them with minimal effort.

I think we need another thread on this topic. How to do a very simple yet effective blind test at home with minimal effort. It has to be fairly easy or it becomes too hard for a normal non-engineer without engineering equipment to do. This is for snake oil debunking tests. Our engineers here online will burrow into the weeds, then the roots and then into soil composition. I have seen everyone fail simple at home, well, at work testing of "is there any difference between these two xyz". This is to debunk the wild claims of the liars, scammers, and hucksters that dominate the high end audio industry.

I like our engineers, without them the technology would not progress. We need their extremely exacting testing for cutting edge research. But, I think for all the non-engineer people on ASR, many just want snake oil debunked as quickly and easily as they can. The tests do not have to be all that hard. If someone says their car can go 100mph in a 1/2 mile every day, all day and any day and your car is crappy because it only does 70, you say ok, lets go. He agrees his car is running great and will have no trouble doing 100 today. You go out to a strip of road and wonder of wonders he cant get his high dollar fancy car to do more than 70mph in a 1/2 mile. You then give him 1 mile and he still can't do it. Then you test your car and it does 70mph too. Well, you just debunked his claim that his will do 100mph any time, any day.

So, there are at least two large groups on ASR. #1 the high tech engineers and #2 the people who want extreme snake oil debunked. The testing for each is radically different. High tech engineer testing is expensive, hard to do correctly and has to factor in many variables and try to reduce those variables to an absolute minimum. Average audio guy wants to know, will this $2000 speaker wire make my system "come alive" and sound like I spent $2000 on two speaker wires? The xyz company says it does. Listeners say it does, but do a simple roughly blind test and total failure. That to me is knowledge that most people want. But it is not what this thread is for. So, we have two groups kind of cross talking a bit. I bowed out when I realized this was getting into serious hardcore science. Still interesting to watch the engineers discuss this though. They are fun to read.

 

[CMOT]

 

[preload]

1) Who is the intended audience?
2) What is the goal of this FAQ?

I'm not sure how this can be written without first establishing #1 and #2.

As predicted, the thread has gone all over the place because the answers to the questions above were never established.

 

[solderdude]

I'm an engineer, not a scientist.
I would be looking to a practical device that would adhere to what scientists think the devices should do.
Then there are software guys that can make software that tells a test device what relays to switch and when and keeps track of responses.
There are also math people involved that tell software people what algos to use for the results.
The practical device has to be operated which should be made easy and foolproof.
Once you have a practical device that can adjust levels switch line and speakers levels (different relays) can do balanced and SE the connected devices also have to be checked for certain properties. That could well be made easy by the software and maybe an ADC could be of help here.
Now you can test several things like mains cords, interlinks, speaker cables, amplifiers (all sorts) but these will require all kinds of different 'adapters' and those adapters should not be suspect in any way.
This would be very costly, time consuming and complex to make and could be helpful to determine all kinds of things.

For these circumstances where many disciplines come together and make something really nice, based on science it would be an idea to fully specify everything that has to be built to pull this of. 1 or 2 may probably get built and could be used for specific research. Nice.
Is the world waiting for this ?



Then you could also write a 'simple' guide to tell you how to 'test' 2 DAC's in a way a bit more 'rigid' and kind of controlled for those that like to look for some truth determined by their own ears. Using stuff that is not purpose built and using items already present. Just describing the methods and things to watch out for.
Maybe ways so some helper could assist and what they should do and what to look out for. How to interpret results obtained.

Another tutorial for amplifiers, one for interlinks, one for speaker cables, one for mains cables... all using stuff they already have.
Thinking about a laptop/tablet/PC with onboard audio for simple comparative measurements and generating test signals.
Maybe suggest some multimeters (cheap to more serious) that could help determine levels.

I think the latter is what most folks would be interested in. If this is the goal someone should/could open a thread only about testing a specific 'component' and make threads for all components separately.
Ideas could be given, discussions made about the procedure and then a guide could be written how to go about it a bit more rigorous and sciency without it becoming impractical and giving better results than incorrectly made comparisons.
No, it can't be proof to others, yes it can still be wrong but at least it is doable and agreed upon.

Not volunteering for this but will put in my thoughts about it.


What would be desirable ?

I think the attempt should be made clear.
What the target audience is and whether it is meant to prove things to others or is only meant for self education.

 

[preload]

You all can design the absolute most rigorous and indisputable blind listening protocol based on the finest switch box, and spend $1million to conduct the perfect experiment. But at the end of the day, if your target audience doesn't understand or believe in the validity of what you're doing, they're still going to go with "their ears" or what their favorite influencer on YouTube tells them is true. Welcome to 2021.

If the purpose of all of this is to debunk beliefs of substantially different sounding power cables, speaker cables, and sota devices like amps and DACs, you're going to have to appeal to people who's minds you're trying to change. Hint: they ain't engineers. All of this techno gibberish won't mean a darn thing to them. Also you all are choosing to fight a difficult to win battle by trying to demonstrate that there is NO audible difference. In my opinion the question should be whether differences are easily or readily audible. That relaxes the methodological requirements and is something the believers might be more inclined to accept from a simple experiment.

 

[audio2design]

As predicted, the thread has gone all over the place because the answers to the questions above were never established.

Bingo

Someone convinced of cable audibility is never going to accept relays in the signal path. People are worried about scientific validity of experiments while making assumptions on component and fixture inaudibility that are target audience may not accept.

Ps chattering relays to disguise a switch presents a distractive masking function that extends the effective switching time as the brain loses then gains back focus. For my own research I have used both relays in soft potting compound and a relay to MOSFET switchover that is then slowly opened or closed while the relay current is reduced softening how loud the switch is. We are talking music not a power supply. Switching at a zero cross in a continuous function is still a discontinuity that leads to unwanted spectral components. These are the things you learn when doing research on the boundaries of human perception.

 

[j_j]

Bingo
Switching at a zero cross in a continuous function is still a discontinuity that leads to unwanted spectral components.

Using only zero crossing, I do agree. A soft sum to 1 fade over even a few milliseconds on very similar inputs (which is what we're talking about) will not do so.

See my post earlier for a file demonstrating this, very clearly.

 

[MarkS]

Bingo

Someone convinced of cable audibility is never going to accept relays in the signal path.

Nor should they. Just swap the damn cables! Or DACs. Or amps. Or whatever. Cover them up so the listener can't see which one is installed.

That's it. You just need an assistant to do the swaps. And you don't need to match volume. Just start every listening session with the volume at zero, and let the listener turn it up (without access to a visual or tactile readout of volume level).

Audiophools never do this because they don't want to know the truth. And no amount of ABX comparator engineering is going to make them want to know the truth.