User spec for a one-person-controlled blind test device

[Rja4000]

Hi there

A lot of discussions occur here about what we can hear as a difference and what we can't hear.
I have my own experience with that.

It's an important question, since measurements should be related to audibility. Ultimately, this is the only thing that matters.

Often, we refer to ABX, or double blind tests.
Unfortunately for us ( I really thing it's unfortunate, since I'm sure most of the users participating here are good will and can be trusted), realizing a real controlled blind test is almost impossible in real life, for most, if not all, of us.
First, you need a second person to operate it.

So what?
Couldn't one design and build an (inexpensive) device that would allow most of us to conduct real controlled tests ?
In my books, that would reinforce the strength of this site. The community is now becoming big enough to be somehow relevant statistically.

Of course, "no significant difference" can hardly be a proof of anything, since you just have to ask your dog/cat/canari to randomly select a source for the result to be irrelevant.
But those who honestly believe they hear a difference would get their chance (and little excuse not to use it).

So, let's start with DAC:
What do we need?

A sort of simplified computer, that will randomly link one of 2 sources to one output, allow control of the sound playback, and record the user's choice, in foobar's ABX plugin way.
An accurate leveling software, which will use some signal, measure DAC output, and accurately match the signal levels.
A switching hardware, controlled by the computer.
A centralized web-based database, to record, aggregate, analyze and show results.

How unrealistic would that be?
Of course, we need to dig deeper in details.

What do you think?
Worth it or not?
Ready to work on it?
What did I miss?

 

[solderdude]

Lets start with the DAC.

You need 2 DAC's to reproduce the same music source at the same time. This is possible.
Then you need to factor in the small timing differences due to different filters etc.
This means you can switch directly because there will be differences when switching from DAC1 to 2 and vice versa.
So switching (or faking switching) must have a small mute of say 0.1s.
It may not be fair to use software volume control to adjust output levels as DAC1 could be on the verge of audible clipping yet not go there when attenuated. Needs bit perfect signal for comparison.
You will also need test tones and a meter as well as potmeters that allow for accurate and small level differences say between 1V and 3V (SE) and balanced to SE conversion if needed that is acoustically transparent before it is sent onward.
You would need software that does random switching or not switching (but always muting) which is either on demand (by pushing a button) or by the software.
You need to log so need to have inputs for that and the software logging and comparing.

Amplifiers and cables are a lot easier to do (but cables is pointless of course)

 

[Rja4000]

Hi
Not being able to use digital volume control (or change source level) for level matching will make this much more complicated.
But why not?
Wouldn't that be representative of a real life case?

I agree that we need a short mute to avoid hearing phase change.

Computer side can be a simple Arduino. But rebuilding a proper bit perfect player is quite some work. So probably easier to build a new ABX plugin for foobar and allow it to control the external system.

Any other thoughts?

We need a computer-controlled physical board with relays to switch sources, too.

 

[solderdude]

Not being able to use digital volume control (or change source level) for level matching will make this much more complicated.
But why not?
Wouldn't that be representative of a real life case?

Using digital volume control in some specific cases could mask certain issues that might be there such as intersample overs.
Also not all digital volume controls let you adjust within 0.1dB.

This is really essential when the goal is a valid (D)BT.

The problem I see is that only a handful of people will ever build buy something like this and after having used it on several well designed DACs the realization kicks in that there are no audible differences to be detected when enough attempts are made (say 20).

This kind of testing is exhausting and boring.

Myself I took the easy route (which has some obvious pitfalls) and compared a few DACs but not high-end just the ones I had at that time.
Recorded the analog output of 3 DACs (24/192) and used RB FLAC well made recordings and ensured peaks would not exceed -3dB.
Then I normalized those recordings (-3dB) and AB'd them (not knowing what was playing).

The results satisfied me personally and could not care less if someone else may or may not pass the same test. For me it is about my personal perception.

 

[Rja4000]

Using digital volume control in some specific cases could mask certain issues that might be there such as intersample overs.

I'm not sure what this is ?

Also not all digital volume controls let you adjust within 0.1dB.

well, we can just adjust the level in the player, can't we? In that case, we can be as accurate as we whish.

The problem I see is that only a handful of people will ever build buy something like this and after having used it on several well designed DACs the realization kicks in that there are no audible differences to be detected when enough attempts are made (say 20).

Yes. That's why it has to be inexpensive.
But how curious are we? How many of us happened to think "and still, I hear a difference..." without having a chance to (dis)prove it?
And how much are costing those high end DACs?
So, if we keep hardware cost below $100€, wouldn't you be interested? Especially if you can use the same for amp comparison.
As for the software, this could be kind of a community effort/funding, as I see it.

Myself I took the easy route (which has some obvious pitfalls) and compared a few DACs but not high-end just the ones I had at that time.
Recorded the analog output of 3 DACs (24/192) and used RB FLAC well made recordings and ensured peaks would not exceed -3dB.
Then I normalized those recordings (-3dB) and AB'd them (not knowing what was playing).
The results satisfied me personally and could not care less if someone else may or may not pass the same test. For me it is about my personal perception.

This has been proposed several times and declined.
I proposed it myself.
I see good reason to decline for DAC differences.
I see less for amplifiers, or anything less "perfect".

 

[solderdude]

There are several posts about intersample overs. just use the search option. It is a problem for a lot of DACs and not an imaginary ones.
The problem goes away when you adjust the volume digitally.
To compare DACs you should be using them bitperfect.

Not all digital volume controls (player or OS) may let you adjust within 0.1dB and again the issues above may go unnoticed.

I have done too much blind tests and fooled myself way too many times sighted. I already know my hearing limits due to many measurements and listening and perception tests. Now that the years start counting I know it will only get worse and never get better.
One does get more experience with listening though.
So I would not be interested in this and would simply DIY it or use a passive volume control and a few switches and let someone else switch.

I know the simple tests I have done recently are not the ultimate ones and used post processing and ADC but they satisfied my needs.
The experiment would not hold up in court nor were expensive DACs and ADCs used.

I rather enjoy listening to music and fooling around with headphones at this point in time so have no interest in designing a PCB for this and integrating Arduino or other controllers.

There might be some guys that would like to build this.