How can DAC's have a SOUND SIGNATURE if they measure as transparent?

[Blumlein 88]

Hello.

I like math and logic and I asked myself the following question:

we take 1000 people totally at random and make them take an 'ABX test' between two DACs, then we repeat this same experiment a dozen times.

We add up the results to find out which of the two devices was unanimously preferred (by listening, of course).

It can be seen from the results that the device chosen is not the one that obtains the best results in the measurements:

- What can we conclude from this ?

- What is important ?

Be objective but above all honest and don't tell me that it's not possible because the measurements say the opposite, DACs have fairly small measurement differences between them nowadays and there is still a very large number of people who are convinced that they hear differences.

My final question is: Are we 100% confident that we are currently measuring everything that should be measured ?

NB: I don't know (what else to measure) but I'm not convinced although I don't question the objective evidence of what we measure nowadays.

PS: I liked the 'intervention' of a member of the forum who said first to listen to a device before proceeding with the different measurements so as not to influence his subjectivity (his impressions) because I have the impression that it is often the opposite that happens quite often...

Regards.

The problem is you are imagining a result. Then asking what we should do. Okay for conjecture. However, the results we do have indicate an extremely low probability of your imagined result occurring. What we should do then is conclude the measurements combined with knowledge of psychoacoustics are fine.

I would note you are unlikely to get unanimous results or you mean something different by that.

If your proposed test with 1000 people doing 12 trials each had an unexpected result you would first want to retrace methodology to see if nothing had contaminated the results. Replication to see if the result was about the same a second time would be one step. We would expect such a test to show correct identification between 11,890 and 12,110 times at the 95% level of confidence that results were random. If for instance 1 device with slightly inferior measurements were preferred 12,050 times it wouldn't mean anything.

 

[Jimbob54]

Hello.

I like math and logic and I asked myself the following question:

we take 1000 people totally at random and make them take an 'ABX test' between two DACs, then we repeat this same experiment a dozen times.

We add up the results to find out which of the two devices was unanimously preferred (by listening, of course).

It can be seen from the results that the device chosen is not the one that obtains the best results in the measurements:

- What can we conclude from this ?

- What is important ?

Be objective but above all honest and don't tell me that it's not possible because the measurements say the opposite, DACs have fairly small measurement differences between them nowadays and there is still a very large number of people who are convinced that they hear differences.

My final question is: Are we 100% confident that we are currently measuring everything that should be measured ?

NB: I don't know (what else to measure) but I'm not convinced although I don't question the objective evidence of what we measure nowadays.

PS: I liked the 'intervention' of a member of the forum who said first to listen to a device before proceeding with the different measurements so as not to influence his subjectivity (his impressions) because I have the impression that it is often the opposite that happens quite often...

Regards.

ABX doesnt test for preference. It tests for correctly recognising differences.

 

[ahofer]

Hello.

I like math and logic and I asked myself the following question:

we take 1000 people totally at random and make them take an 'ABX test' between two DACs, then we repeat this same experiment a dozen times.

We add up the results to find out which of the two devices was unanimously preferred (by listening, of course).

It can be seen from the results that the device chosen is not the one that obtains the best results in the measurements:

- What can we conclude from this ?

- What is important ?

Be objective but above all honest and don't tell me that it's not possible because the measurements say the opposite, DACs have fairly small measurement differences between them nowadays and there is still a very large number of people who are convinced that they hear differences.

My final question is: Are we 100% confident that we are currently measuring everything that should be measured ?

NB: I don't know (what else to measure) but I'm not convinced although I don't question the objective evidence of what we measure nowadays.

PS: I liked the 'intervention' of a member of the forum who said first to listen to a device before proceeding with the different measurements so as not to influence his subjectivity (his impressions) because I have the impression that it is often the opposite that happens quite often...

Regards.

This is as close as you are going to get. And it’s actually one of the rare positive results:

"High-End" DAC Blind Listening Results - PART I: Devices Unveiled!

A blog for audiophiles about more objective topics. Measurements of audio gear. Reasonable, realistic, no snakeoil assessment of sound, and equipment.

archimago.blogspot.com

I’ve tried to inventory others in this thread, but it is still incomplete. The link in the first post is to a great compilation, but there’s been a lot of ‘link rot’.

Catalogue of blind tests

I thought this thread on Head-Fi was a valiant effort to put together blind tests that have been performed over the years. Given that there have been many more (I'm thinking Archimago's tests, among others), it would be fun to open source a complete list...

audiosciencereview.com

 

[ICIETDIYEUR]

Hello.

'Of course', I was expecting this kind of answer...

Some people call it self-persuasion, I call it 'reassurance' in a way.

Just one last question: Would you buy a device just because it gets good measurements without even hearing it ?

 

[ahofer]

Would you buy a device just because it gets good measurements without even hearing it ?

Not sure if you were responding to me, but if we are talking about amps, streamers, and DACs the answer is not only *would* I, but I *have*. Also, the last time I bought speakers I limited my search to those that had satisfactory spinorama results, and only auditioned those that did. In doing so, I found I prefer the wider dispersion speakers over narrower (Revel>KEF, although they are both satisfying to listen to for long periods of time). In the future, if I take time to go listen to speakers, it will only be if the measurements are solid. If someone wants to audition poorly-measuring speakers while I’m there, that’s fine, but they aren’t worth my valuable time.

I bought two Class D amps (March, NAD), RME, WiiM, FiiO DACs, and a topping headphone amp on measurements alone. I have two Raspberry Pi computers acting as streamers and I bought those not only without listening, but without measurements!

I’m contemplating buying a pair of Dan Clark headphones…on measurements alone!

 

[Mart68]

Hello.

'Of course', I was expecting this kind of answer...

Some people call it self-persuasion, I call it 'reassurance' in a way.

Just one last question: Would you buy a device just because it gets good measurements without even hearing it ?

I call it making decisions based on facts and evidence.

For electronics yes, always buy without 'listening' to them - how do you listen to an electrical voltage in a circuit? It's not a piano or guitar - which I would audition before buying.

 

[ahofer]

It's not a piano or guitar - which I would audition before buying

Well, in that case we have the all-important “feel” aspect of the instrument, and the fact that its idiosyncrasies (‘poor/aberrant measurements’) are part of its sonic signature.

Although with guitars in stores, not all of them are well set-up, and that can put you off a good instrument. Ideally, the whole store inventory would be properly set up (and maintained) before display. I usually grab the arch tops and semi-hollows and I can’t tell you how many awful instrument set-ups I encounter. Similarly, some very cheap instruments will be amazing if you give them a fret job and a set-up.

 

[Sir Sanders Zingmore]

Hello.

I like math and logic and I asked myself the following question:

we take 1000 people totally at random and make them take an 'ABX test' between two DACs, then we repeat this same experiment a dozen times.

We add up the results to find out which of the two devices was unanimously preferred (by listening, of course).

It can be seen from the results that the device chosen is not the one that obtains the best results in the measurements:

- What can we conclude from this ?

- What is important ?

Be objective but above all honest and don't tell me that it's not possible because the measurements say the opposite, DACs have fairly small measurement differences between them nowadays and there is still a very large number of people who are convinced that they hear differences.

My final question is: Are we 100% confident that we are currently measuring everything that should be measured ?

NB: I don't know (what else to measure) but I'm not convinced although I don't question the objective evidence of what we measure nowadays.

PS: I liked the 'intervention' of a member of the forum who said first to listen to a device before proceeding with the different measurements so as not to influence his subjectivity (his impressions) because I have the impression that it is often the opposite that happens quite often...

Regards.

If instead, the results of the above experiment showed no statistically significant difference, what would you say to the “very large number of people who are convinced that they hear differences.”

Be objective and above all honest.

 

[solderdude]

ABX doesn't test for preference. It tests for correctly recognizing differences.

Yep and when done right it is very exhausting to do and one should not do this for more than say ... 10 to 15 minutes.

Really ... this AB(X) when the differences are at or nearing inaudible are very tedious.
Besides... when someone plays recordings of a few DACs with very little differences in performance on gear that performs much worse what are people really listening to ?

I'm a huge proponent for blind testing but having done a few I know how hard that can be and how important listening levels and the used gear is.

 

[Mart68]

. Similarly, some very cheap instruments will be amazing if you give them a fret job and a set-up.

Yes that's true. I had a pro guitar tech strip and set up my cheapo Les Paul copy and it played like a real Les Paul - for a few hours. Then it quickly went bad again.

 

[oleg87]

Hello.

'Of course', I was expecting this kind of answer...

Some people call it self-persuasion, I call it 'reassurance' in a way.

Just one last question: Would you buy a device just because it gets good measurements without even hearing it ?

It’s a piece of electronic equipment, not a headphone or something where measurements aren’t really capable of capturing all meaningful aspects of real world performance, comfort, etc. I’m no less susceptible to sighted bias than anyone else, so without controls it wouldn’t be a very informative exercise anyway.

 

[Somafunk]

Just one last question: Would you buy a device just because it gets good measurements without even hearing it ?

I’ve never listened to any of my equipment before buying it, everything I own was bought online.

 

[pkane]

Hello.

'Of course', I was expecting this kind of answer...

Some people call it self-persuasion, I call it 'reassurance' in a way.

Just one last question: Would you buy a device just because it gets good measurements without even hearing it ?

I've purchased a good number of audio devices based on measurements. I've been happy with all my purchases.

What's more, when a good-measuring device doesn't sound right in my system, that tells me that I need to investigate what I did wrong in configuring, connecting, setting it up, or positioning it. It's a liberating experience, since now I have a predictable, consistent way to investigate what is wrong if something doesn't meet my expectations. It's completely the opposite of what I used to do -- continuously swapping random components, trying out various tweaks, and frequently being disillusioned by pricy high-end devices recommended by others that just didn't work or didn't do anything better.

 

[ICIETDIYEUR]

Hi.

I am delighted that the subject is a real discussion with different unclear-cut points of view and visibly open to discussion and sharing.

So I have a question (which I have already asked 'indirectly') of which I would like to have the opinions:

Do you think that the measurements that are currently being carried out are 'sufficient', well interpreted in the context of our current knowledge, but above all, are we not forgetting 'something' that could fall within the scope of the measurements to be carried out but that has escaped us in our good understanding of the capabilities of a device to reproduce 'sound' with the best possible fidelity ?

My question is very serious, it is not a question of 'mockery', or of having a positive or negative position in advance but just, I hope, will perhaps serve to make us discover something new to make further progress.

Thank you for taking your time to read me.

Regards.

 

[solderdude]

Hi.

I am delighted that the subject is a real discussion with different unclear-cut points of view and visibly open to discussion and sharing.

So I have a question (which I have already asked 'indirectly') of which I would like to have the opinions:

Do you think that the measurements that are currently being carried out are 'sufficient', well interpreted in the context of our current knowledge, but above all, are we not forgetting 'something' that could fall within the scope of the measurements to be carried out but that has escaped us in our good understanding of the capabilities of a device to reproduce 'sound' with the best possible fidelity ?

What other signal is there than 1 (or 2 for stereo) voltage(s) changing amplitude over time and the relation between the input and output signals as well as between channels (for stereo) ?

Thus.. all that needs to be measured is voltage (into a load) over time for those 1 (or 2) signals.
This can be done with the a resolution that greatly exceeds all human and non human accuracy.
All that is left is to analyze that acquired data and make results visible in a comprehensive way.

My question is very serious, it is not a question of 'mockery', or of having a positive or negative position in advance but just, I hope, will perhaps serve to make us discover something new to make further progress.

My answer was equally serious. In audio the biggest issue is perception as that has a poor relation with signal fidelity above (and below) certain limits (thresholds).
This requires the complex field of psycho-acoustics.

Measurements get trickier when the electrical signal is converted in transducers to mechanical movement which moves air.
When moving air hits the ears and neurons are firing in the brain it gets more complicated.

Fortunately electronics from transducer signal (microphone/pickup etc) right up to the driving of another transducer (speaker/headphone) is all electric and fully measurable.

Thank you for taking your time to read me.

Regards.

 

[Newman]

I am delighted that the subject is a real discussion with different unclear-cut points of view and visibly open to discussion and sharing.

I think you are misrepresenting the discussion, in a manner that creates an impression that there is debate where there is none.

All we really have are facts, acknowledgement of facts, unawareness of facts, and denial of facts.

So I have a question (which I have already asked 'indirectly') of which I would like to have the opinions:

Do you think that the measurements that are currently being carried out are 'sufficient', well interpreted in the context of our current knowledge, but above all, are we not forgetting 'something' that could fall within the scope of the measurements to be carried out but that has escaped us in our good understanding of the capabilities of a device to reproduce 'sound' with the best possible fidelity ?

Ask your question to yourself, the other way around: Do we currently have evidence from controlled listening tests that DACs and amplifiers, with not-completely-broken distortion levels and flatness of frequency response that exceeds commonly-held understanding of the limits of human hearing acuity, passing well-conducted ABX tests for detection of difference?

The answer seems to be that we do not.

And on that basis, ask yourself another question: taking that as the best available evidence, what can we conclude as a working hypothesis on the matter of whether we are measuring sufficiently right things for DACs and amplifiers? The answer seems to be that we are.

Your questions are in fact directed to the nature of science itself, because they are a hypothetical about data that does not exist, and you are asking whether science and its followers will learn if that data arrives one day. The answer is that they will, just as they already have when confronted with 'ground-breaking' data in the past.

The reason that you and others of the same ilk, with your unoriginal questions, keep asking these questions, is because you cannot believe that your sighted (uncontrolled) listening experience is so grossly deceiving you with respect to the attributes of the sound waves themselves. Well, the results are in and the gross deception is real. Human perception is highly deceptive when given the opportunity, to a degree that is counter-intuitive.

The challenge is not whether science and its adherents will learn from new and challenging data: the challenge is for the layperson to accept that sighted listening perceptions cannot be trusted.

cheers

 

[oleg87]

Do you think that the measurements that are currently being carried out are 'sufficient', well interpreted in the context of our current knowledge, but above all, are we not forgetting 'something' that could fall within the scope of the measurements to be carried out but that has escaped us in our good understanding of the capabilities of a device to reproduce 'sound' with the best possible fidelity ?

Well, you measure what you are set up to measure. If you are testing 1khz SINAD, you aren't going to see how much headroom a DAC has for intersample overs. Measurements can't capture every permutation of external conditions like loading or whether some cheap laptop's USB power is way off-spec. Amir or whoever's doing the testing could miss implementation bugs in certain modes or conditions, etc.

I've seen zero good evidence of something producing an audible effect that *isn't* easily measurable, but you need to actually measure it. What we *do* know is that there's a very robust mechanism for fictious differences in human hearing, so if we don't have reason to suspect an amp or dac has problems the usual panel of tests isn't showing, some guy hearing something really isn't sufficient evidence of a problem existing.

 

[VientoB]

I’ve never listened to any of my equipment before buying it, everything I own was bought online.

Best way.

 

[ICIETDIYEUR]

Thank you for your answers.

According to my personal reflection, I think and I understand much better then why Bruno PUTZEYS has moved on to the study of speakers and loudspeakers has obviously it seems that we have reached 'limits' regarding the capabilities of amplifiers and DACs nowadays.

It seems obvious to me then that this should be the way to be favored now by manufacturers and brands without constantly inundating us with products whose 'only' difference or absolute novelty is the reference or the name of the model that is written on it

 

[Newman]

It seems obvious to me then that this should be the way to be favored now by manufacturers and brands without constantly inundating us with products whose 'only' difference or absolute novelty is the reference or the name of the model that is written on it

Yes, but only on the assumption that everyone loses interest in sighted listening impressions as a basis for purchase.

As of today, that is very, very, very, very far from reality. So manufacturers will respond to reality.