How to ensure the exhaustiveness of measurements?

[JustAnAudioLover]

Hi there!

I've been browsing this forum for quite a bit of time now (didn't have an account at the time). I've read numerous reviews from @amirm, and learnt a ton of things about DACs, amplifiers, and snake oil. Saved so much money not trying to buy the new super shiny amplifiers for my speakers / headphones / earbuds when you can actually some really good ones for cheap.

I have a question though, and it's about measurements. I've watched @amirm's video on how to do measurements (controlled tests, etc.) and read quite many articles on the subject. But, I'm still not sure about one thing:

How can we be sure that measurements are exhaustive?

Or, put another way, how can we be sure that all the measurements performed on, say, a DAC, like SINAD, jitter and so on are complete enough to say that a device is basically transparent to the ear? How can be sure that there isn't another thing that impacts the sound and that we didn't measure, maybe because science hasn't progressed enough to even consider the very existence of that thing?

For instance, let's say you're measuring room acoustics. You want to know how your room sounds. You measure the frequency response using a single position. Let's say you get a nice flat line. So you say, "well it's perfect, I've got a perfect room!". But actually, if you measure at another position, you may end up finding out that you have a huge dip in the high frequencies, but as the sound bounces on your walls it is reflected back and fills in that dip. So the sound you're actually getting comes at least partially from reflections and have a negative impact on the sound. But you don't see it with your 1-position FR graph, and you didn't know better because you weren't aware that you needed to measure something else to get the full picture.

Now, I'm not downplaying @amirm's work, from everything I've read he clearly knows his stuff, way more than almost anyone else in that field, and I'm not criticizing anything he's done. But I'm curious - how can we be sure there's not some measurements missing to get the full picture?

Thanks in advance for those who will enlighten me on that matter

(P.S. english is not my native language, so sorry in advance if i made some mistakes ^^")

 

[amirm]

How can we be sure that measurements are exhaustive?

Should that be the goal? I can certainly make it look exhaustive by running a hundred test. Much of it would be all redundant to what we are measuring.

Now, if someone hears something that we can't explain from measurements, sure. But that case has yet to come up.

Our testing easily distinguishes performance of countless products. So clearly it has high discrimination. Perfection doesn't exist in life, nor here.

 

[Oddball]

I find the tests at this site amazing and enlightening.

If I ever dared to ask for more, it would be compression tests for the big towers that claim 115dB or so SPL. Having them tested at 96dB for distortion is for sure extremely helpful, and goes beyond what other sites do, but checking what they can do above that would be interesting to understand how they perform at the max specs they are supposedly rated to. Probably not many that want to test 96dB limit, but peaks can get above that in both stereo and HT if one gets playful.

 

[Blumlein 88]

In regard to your speaker example, study the work of Toole and Olive on speakers. Perhaps purchase Floyd Toole's book (I think a new 4th edition is about to come out). You especially get real issues in the lower frequencies with speaker position and listening positions say 300 hz and lower. Speakers with a controlled off axis directivity will work best for most people from testing that has been done.

 

[JustAnAudioLover]

Should that be the goal? I can certainly make it look exhaustive by running a hundred test. Much of it would be all redundant to what we are measuring.

Now, if someone hears something that we can't explain from measurements, sure. But that case has yet to come up.

Our testing easily distinguishes performance of countless products. So clearly it has high discrimination. Perfection doesn't exist in life, nor here.

That's a very interesting response, thanks

 

[oleg87]

"Exhaustive" is a pretty high bar. Unless you're designing something that has to be failsafe or people die, at some point you have to limit yourself to the plausible.

You should understand the operation of the device in question, how things can go screwy, and what needs to be exercised. People have been designing amplifiers for well over a century for example, so there's a pretty large body of technical literature on the matter, and ultimately what matters most is the performance in the range of frequencies humans can hear so that's what's usually focused on.

 

[Jim Taylor]

To me, tests are exhaustive 1) when I look at them and cannot say that the published results (or lack thereof) might be hiding something, and 2) I cannot find a pertinent characteristic for which there is no answer. ("Pertinent" implies real, rather than imagined.)

Tests for electronics are more easily displayed than tests for speakers, but with the advent of the Klippel NFS, even test results for speaker characteristics can be more easily managed ... although it's still no piece of cake.

 

[BDWoody]

Or, put another way, how can we be sure that all the measurements performed on, say, a DAC, like SINAD, jitter and so on are complete enough to say that a device is basically transparent to the ear?

A convincing answer isn't going to given in one or two sentences. The body of knowledge and body of evidence that continues to build supports this conclusion. It is based on both an understanding of circuits and electronics, and psychoacoustics, and supported by about 100 years of research and evidence.

How can be sure that there isn't another thing that impacts the sound and that we didn't measure, maybe because science hasn't progressed enough to even consider the very existence of that thing?

The lack of anything but anecdotal claims and misleading propaganda for a start.

Don't you believe that Danny or Paul or Mikey or Rob or Darko or any of them would actually go through the hassle of putting together a rigorous blind test and publish it if it would actually support their claims?

Can you imagine if they got some magic box or magic cable or power conditioner where a statistically significant number of participants expressed a preference that could be consistently shown and repeated? They would change the audio world, and likely sell thousands of units just to our membership.

If someone wants to claim we must be missing something, they simply have to provide evidence.

They won't, because they can't, and they either know it or are too afraid to put their Golden Ear reputation to the test.

 

[Blumlein 88]

Don't you believe that Danny or Paul or Mikey or Rob or Darko or any of them would actually go through the hassle of putting together a rigorous blind test and publish it if it would actually support their claims?

They want this so badly, some of them have pretended they have done just that only they didn't really do it rigorously or in one case at all. Just described something else as this.

 

[Travis]

Hi there!

I've been browsing this forum for quite a bit of time now (didn't have an account at the time). I've read numerous reviews from @amirm, and learnt a ton of things about DACs, amplifiers, and snake oil. Saved so much money not trying to buy the new super shiny amplifiers for my speakers / headphones / earbuds when you can actually some really good ones for cheap.

I have a question though, and it's about measurements. I've watched @amirm's video on how to do measurements (controlled tests, etc.) and read quite many articles on the subject. But, I'm still not sure about one thing:

How can we be sure that measurements are exhaustive?

Or, put another way, how can we be sure that all the measurements performed on, say, a DAC, like SINAD, jitter and so on are complete enough to say that a device is basically transparent to the ear? How can be sure that there isn't another thing that impacts the sound and that we didn't measure, maybe because science hasn't progressed enough to even consider the very existence of that thing?

For instance, let's say you're measuring room acoustics. You want to know how your room sounds. You measure the frequency response using a single position. Let's say you get a nice flat line. So you say, "well it's perfect, I've got a perfect room!". But actually, if you measure at another position, you may end up finding out that you have a huge dip in the high frequencies, but as the sound bounces on your walls it is reflected back and fills in that dip. So the sound you're actually getting comes at least partially from reflections and have a negative impact on the sound. But you don't see it with your 1-position FR graph, and you didn't know better because you weren't aware that you needed to measure something else to get the full picture.

Now, I'm not downplaying @amirm's work, from everything I've read he clearly knows his stuff, way more than almost anyone else in that field, and I'm not criticizing anything he's done. But I'm curious - how can we be sure there's not some measurements missing to get the full picture?

Thanks in advance for those who will enlighten me on that matter

(P.S. english is not my native language, so sorry in advance if i made some mistakes ^^")

Your English is great. In the area of speakers, follow the advice given in a previous answer, get Floyd Toole's latest version of his book. That is going to be enough for 98% of people, but for some, you need to join AES so you have all of the latest papers on speakers, measurements, etc. It is there you will what what the official position of AES on what areas need further investigation on some areas of measurement with loudspeakers, and where they are at on that.

For amplifiers and DACs, I highly recommend reading this entire thread. This really, really helped me to understand why Amir's dashboard is the way it is, why these measurements, out of the 100s that can be done have been chosen, etc. In addition, the thread, for me, explained what measurements have been backed up with papers, and science. Other measurements are logical extensions of things supported by testing and science. Nice community effort to explain, in a practical way, what the numbers mean (if anything).

Audibility thresholds of amp and DAC measurements

Introduction A recurring theme on ASR is whether or not the various measured qualities of the devices are audible. In this post, I'll present some clear and visual thresholds for when those imperfections can be considered a potentially audible concern. I will not explain the basics of amp/DAC...

www.audiosciencereview.com

 

[Keith_W]

There are certainly criticisms from other parties about things that Amir does not test, for example by John Siau. I have also heard criticisms from amplifier designers about "things not tested on ASR" - I can't seem to find those right now.

To be honest I have very little interest in hardware tests, so I don't understand what they are criticising or why it is important. The low hanging fruit in every system is rooms, speakers, and DSP - ergo what is important are acoustic measurements, and then understanding acoustics and psychoacoustics - so that's what I focus on.

My first question whenever someone criticises Amir for something he did not test is - "is it audible?". Sometimes they will insist till they are blue in the face that it is indeed audible, but if they are trying to explain why their power cord sounds better I usually ignore them because they are probably doing a Gish gallop. But sometimes the person doing the criticism is someone credible, like John Siau. In which case I decide it's Amir's problem, not mine.

 

[wwenze]

Class D and 200kHz noise-free bandwidth: John Siau vs ASR forum

 

[Mart68]

For instance, let's say you're measuring room acoustics. You want to know how your room sounds. You measure the frequency response using a single position. Let's say you get a nice flat line. So you say, "well it's perfect, I've got a perfect room!". But actually, if you measure at another position, you may end up finding out that you have a huge dip in the high frequencies, but as the sound bounces on your walls it is reflected back and fills in that dip. So the sound you're actually getting comes at least partially from reflections and have a negative impact on the sound. But you don't see it with your 1-position FR graph, and you didn't know better because you weren't aware that you needed to measure something else to get the full picture.

You're right about that but it's already known - in room measurements won't necessarily give you a correlation with what you hear once above the transition frequency/band. The solution is to use the anechoic measurements as a guide to what the speaker is doing at those frequencies.

It's a complex subject so some familiarity with the basics has to be assumed or every speaker review would need a ton of explanatory text.

 

[MaxwellsEq]

How can we be sure that measurements are exhaustive?

You have had some excellent answers. It's worth recalling that humans have been measuring audio and audio electronics for more than a century with increasing precision.

Over to you - what domestic audio reproduction experience is not being properly measured and why?

 

[JustAnAudioLover]

You have had some excellent answers. It's worth recalling that humans have been measuring audio and audio electronics for more than a century with increasing precision.

Over to you - what domestic audio reproduction experience is not being properly measured and why?

I'm not saying that measurements aren't done properly. Again, as stated in my initial post, I'm not criticizing @amirm's work. I'm wondering on a more "high-level" point of view, how to ensure that measurements are exhaustive enough that we can say, from the results we've got, that a DAC or amp is indeed acoustically transparent to the human ear.

Answers have been very interesting so far, it gives me a better understanding on how that works

 

[MaxwellsEq]

I'm not saying that measurements aren't done properly. Again, as stated in my initial post, I'm not criticizing @amirm's work

You misunderstood me. I did not suggest or imply that you were criticizing Amir or anyone else. Nor do I suggest or imply that you were saying existing measurements were done poorly.

I'm wondering on a more "high-level" point of view, how to ensure that measurements are exhaustive enough that we can say, from the results we've got, that a DAC or amp is indeed acoustically transparent to the human ear.

So, again, I'm asking you - in your opinion, what domestic audio reproduction experience is not being measured?

 

[JustAnAudioLover]

You misunderstood me. I did not suggest or imply that you were criticizing Amir or anyone else. Nor do I suggest or imply that you were saying existing measurements were done poorly.

So, again, I'm asking you - in your opinion, what domestic audio reproduction experience is not being measured?

I don't get what you're asking. Nothing is missing in my opinion, and that's precisely why I made this thread in the first place. The fact I don't see any missing measurement doesn't mean there isn't any missing one.

 

[JSmith]

How can we be sure that measurements are exhaustive?

Or, put another way, how can we be sure that all the measurements performed on, say, a DAC, like SINAD, jitter and so on are complete enough to say that a device is basically transparent to the ear? How can be sure that there isn't another thing that impacts the sound and that we didn't measure, maybe because science hasn't progressed enough to even consider the very existence of that thing?

I don't get what you're asking.

?


JSmith

 

[JustAnAudioLover]

?


JSmith

What's confusing you? I'm confused by the fact you seem to be confused ^^"
Maybe I didn't explain something properly?

In my initial post I say that I want to know how we can ensure if the measurements done here are enough to call a device "acoustically transparent".
Then I'm asked what measurement I think is missing. That's not the point, I'm not aware of any missing measurement, but that doesn't mean there isn't any missing one. When I first came here I didn't even know what SINAD was and why it was important for instance.

 

[sergeauckland]

As to amplifiers or electronics generally, there are only three measurements that affect what we hear. Distortion, noise and frequency response. Do those at various power levels and permitted loads, and that will fully characterise the amplifier.

Loudspeakers will need additional measurements, polar response and pair-matching for stereo for example, but again, these measurements will characterise what a loudspeaker can do.

Audio has been measured for 100 years, so the body of knowledge is pretty substantial.

There's no evidence that any measurements are lacking that would affect what we hear.

S