Master Thread: Are Measurements Everything or Nothing?

[amirm]

Is it true that using a slow filter gives better SINAD than a sharp filter?

In general it is the other way around. Look at my THD+N tests versus frequency that go up to 20 kHz:

The red diagonal line is the result of a very slow filter. The measured THD+N shoots so high it is off the charts. The high sample rate in green/brown lines eliminates the effect of the filter and shows much lower THD+N.

The reason for the rise is not that the DAC is more distorted but that the spurious components getting through due to slow filter get added into what makes up THD+N and inflates that.

 

[Sharur]

In general it is the other way around. Look at my THD+N tests versus frequency that go up to 20 kHz:

The red diagonal line is the result of a very slow filter. The measured THD+N shoots so high it is off the charts. The high sample rate in green/brown lines eliminates the effect of the filter and shows much lower THD+N.

The reason for the rise is not that the DAC is more distorted but that the spurious components getting through due to slow filter get added into what makes up THD+N and inflates that.

This is the post which brought up my question.

 

[Sharur]

Also, have any successful voltage matched blind tests been documented between an Apple dongle and any other well measuring DAC? The question this thread asks is whether we were all wrong about SINAD. However, I haven't seen anyone get close to the "audibly transparent" IMD threshold (http://www.klippel.de/listeningtest/?page=instructions) with real music.

 

[_thelaughingman]

I am going to hyperlink this thread to every response that happens to question validity of objectivity and measurements in any of the threads. Enough is enough as you said.

 

[Digby]

@amirm I'd like to know your position on the question. I imagine you are somewhere in the middle or else why bother with subjective listening impressions in your reviews? In a number of these listening reviews you have said how you couldn't really hear a fault that you measured and that you weren't always sure whether EQ improved a dip/peak or not.

Sometimes you was surprised by how much you enjoyed a speaker, I think the JBL 4349 was one of these, even though the measurements were far from the best you had seen. It seems the speakers provided something in reproduction that the measurements themselves didn't make obvious, which would suggest that measurements (at least as they currently stand) are not the be all and end all, at least when it comes to enjoying music.

Your thoughts?

 

[restorer-john]

@amirm This goes to the suggestion I made years ago.

Do a review thread, present it and lock it. Only make changes where there were errors or omissions.

Have a companion discussion thread with a link from the review thread. If it goes off the rails it can get moderated, locked, whatever and unlinked/deleted without poisoning the main review thread. The discussion threads will die down eventually and the review threads will still be clean, full of facts and a useful resource.

I don't agree with attempting to sanitise discussion on particular review products or your testing methodology by steering people to a catch all, out of the way thread. It isn't a good look at all IMO. And, it won't work anyway- people will still discuss whatever they want and you will only give the moderators a whole lot more work.

 

[amirm]

@amirm I'd like to know your position on the question. I imagine you are somewhere in the middle or else why bother with subjective listening impressions in your reviews? In a number of these listening reviews you have said how you couldn't really hear a fault that you measured and that you weren't always sure whether EQ improved a dip/peak or not.

Sometimes you was surprised by how much you enjoyed a speaker, I think the JBL 4349 was one of these, even though the measurements were far from the best you had seen. It seems the speakers provided something in reproduction that the measurements themselves didn't make obvious, which would suggest that measurements (at least as they currently stand) are not the be all and end all, at least when it comes to enjoying music.

Your thoughts?

I have a scale for how much measurements matter for each category of products:

DACs: 100%
Amplifiers (headphone and speaker): 80 to 90% due to variability of available power. Hard to internalize how much power is available/enough without listening tests.
Speakers: 70 to 80%
Headphones: 50 to 80% (measurements too variable)

This is why you see me do listening tests for the last two categories and half of second (headphone amps).

 

[amirm]

@amirm This goes to the suggestion I made years ago.

Do a review thread, present it and lock it. Only make changes where there were errors or omissions.

Member questions/comments help make the reviews better. Putting them elsewhere makes it hard for members to reference what is in the original review.

 

[amirm]

I don't agree with attempting to sanitise discussion on particular review products or your testing methodology by steering people to a catch all, out of the way thread.

If the comment is specific to the product being reviewed, then it is fine to be asked and discussed there. This thread is for the generic comments that apply to any review we do here.

 

[_thelaughingman]

Can we expand on this further and make this a sticky somewhere.

Guidelines to engage in meaningful conversations on ASR

- Engage in a topic from the perspective of wanting to learn something new about the hobby of being an audiophile.
- Contribute to the topic/thread with good knowledge that has some semblance of support in objective measurements or proven science.
- Disengage from making statements or claims that are steeped in subjective opinion. ie: a transparent dac at $5k is better sounding than a dac at $150. Etc

 

[amirm]

That's a very good suggestion. Maybe we quote your list anytime the topic is brought up.

 

[gallantus]

Alright, I will kick this off!!! Thanks @amirm for setting this up.

First, I believe in measurements. Every gear I buy I need to look at measurements if available. I own the Topping D90SE, because it measures the best, the reason why I brought it, amongst the functionality and the fact that it was on sale. However, I didn't keep it because it measures the best, because I thought it sounds terrific. I am a trained EE, albeit non practicing. As an engineer, I say, you need to measure it, in order to build it.

Also let me get out of the way, there are legit snake oil out there and you know what I am talking about.

But is measurement for electronics the end all be all to guage sound quality?

If two amps measured exactly alike, but they are from different manufacturers with different design, will they sound exactly alike?

Can one buy an amp or DAC, just by looking at the measurements without any audition and know that it will sound like any other amp or DAC only less or more distortion?

If so, why are there so many people not have the Benchmark AHB2 and LA4 and the Topping D90SE combination (budget and use case functionality aside). After all, they measured the best. Yes, yes, some will say because, I need more power (you can bridge the Benchmark into mono btw), some will say I need some other functionalities, etc etc.

Why does x SINAD, y THD and z SNR matter when your ears likely can't tell the difference once you hit a certain threshold? Is it for bragging rights? A peace of mind?

These are honest questions, maybe a bit exoteric and philosophical, but honest questions nevertheless.

 

[gallantus]

I have a scale for how much measurements matter for each category of products:

DACs: 100%
Amplifiers (headphone and speaker): 80 to 90% due to variability of available power. Hard to internalize how much power is available/enough without listening tests.
Speakers: 70 to 80%
Headphones: 50 to 80% (measurements too variable)

This is why you see me do listening tests for the last two categories and half of second (headphone amps).

This is super helpful, I don't think I have seen your position on this anywhere (or at least my first time seeing it). Your position answered a lot of my questions

 

[Madjalapeno]

Headphones: 50 to 80% (measurements too variable)

Do you think part of that is that human anatomy varies so much compared to the test fixtures?

 

[BoredErica]

2 cents... Some would say they're worth even less than that but here it goes...

5. You all must not listen to music at all.

Subjectivists bring this up to take a jab at objectivists but it's so absurd I don't know how to respond to it. What do subjectivists think objectivists actually do after they make fancy graphs and spend a ton of money on their system? They just let the gear sit there? Make endless graphs forever? I listen to my speakers every day! Gotta get my money's worth.

3. I trust my ears, not graphs.

Human brains are incredibly fallible. Placebo is real and that's why medical trials control for it. I'm glad when I board an airplane the airplane wasn't built based off of intuition alone, or the drugs I take tested by giving them to the scientists friends and family with their effectiveness judged based on their feedback. People want to believe they have an entire hobby going on here where everything sounds different. It's cool and people have bought into it because a bunch of people who have no business teaching others about audio said some amps are more 'musical' than other amps or the Skin Effect ruins audio cables. I'm more interested in how reality actually works rather than what makes me feel good.

Ironically it's the objectivists who trust their ears. I trust my ears, so I'd like only my ears tested whenever possible. I don't need my eyes to tell me what I'm listening to. It's the subjectivists who don't trust their ears. So many self proclaimed audio gurus all of a sudden fail to spot differences between two transparent pieces of gear when they can't see what they're listening to.

I don't listen to graphs. I listen to music.

And how that music sounds is based on real things that are described in graphs.

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.

I can't even use the word 'double blind testing' in most areas of Headfi and most of audio spaces online be it Youtube or forums are dominated by subjectivists, so it's no surprise the few spots for objectivists don't take too kindly to subjectivists when they cause others to waste their money on things that don't make a difference, encouraging others to peddle yet more snake oil to unsuspecting masses.

6. Why don't you all buy the best SINAD gear?

I don't have infinite money. I care about how products look and various features. SINAD is not the end all be all of audio measurements, otherwise Amir and others can save a whole lot of time ignoring every other test.

--
Even for people adamant that they like a certain colored sound, knowing what it is they like allows them to shop better or EQ whatever they like in better. Knowing how reality works makes things more complicated yes, but also empowers you to do things you otherwise would not have been able to.

Also for people who aren't quite as much of an objectivist as I am, there is room for grey area. You don't have to think 'all of my subjective feelings about this headphone are meaningless'. You could instead think 'well I know how headphones sound is not an entirely 100% settled on measurements on a standard head alone since it can vary based on individual things like head size and earcup seal, so my subjective thoughts might be meaningful, though I still take them with a grain of salt given what I know about how fallible human brains are.' Of course, it only makes sense to have a grey area filled in with subjective opinions on how things sound when it comes to areas where measurements alone are not enough. Where measurements are enough to describe how something sounds, subjective opinions are worthless.

 

[abdo123]

Alright, I will kick this off!!! Thanks @amirm for setting this up.

First, I believe in measurements. Every gear I buy I need to look at measurements if available. I own the Topping D90SE, because it measures the best, the reason why I brought it, amongst the functionality and the fact that it was on sale. However, I didn't keep it because it measures the best, because I thought it sounds terrific. I am a trained EE, albeit non practicing. As an engineer, I say, you need to measure it, in order to build it.

Also let me get out of the way, there are legit snake oil out there and you know what I am talking about.

But is measurement for electronics the end all be all to guage sound quality?

If two amps measured exactly alike, but they are from different manufacturers with different design, will they sound exactly alike?

Can one buy an amp or DAC, just by looking at the measurements without any audition and know that it will sound like any other amp or DAC only less or more distortion?

If so, why are there so many people not have the Benchmark AHB2 and LA4 and the Topping D90SE combination (budget and use case functionality aside). After all, they measured the best. Yes, yes, some will say because, I need more power (you can bridge the Benchmark into mono btw), some will say I need some other functionalities, etc etc.

Why does x SINAD, y THD and z SNR matter when your ears likely can't tell the difference once you hit a certain threshold? Is it for bragging rights? A peace of mind?

These are honest questions, maybe a bit exoteric and philosophical, but honest questions nevertheless.

Seems like you figured out all the answers already.

the only thing surprising here is that you say you bought the D90SE on sale, which is very surprising tbh.

you should never do that to a flagship product, it brings its market value down. Now people will only buy it at its discounted price.

 

[gallantus]

Seems like you figured out all the answers already.

the only thing surprising here is that you say you bought the D90SE on sale, which is very surprising tbh.

you should never do that to a flagship product, it brings its market value down. Now people will only buy it at its discounted price.

I be damn shocked if no one else saw/know about this deal. In fact, this deal came back on for Black Friday. I brought it knowing that it would be on sale for 11/11 and the vendor had a 30 day price match policy after purchase. So I got it early and got the sale price.

I also have the LKS MH-DA004 on sale.

And not sure what you meant I figured out all the answers already. These are honest questions and up for members here to discuss.

 

[abdo123]

I be damn shocked if no one else saw/know about this deal. In fact, this deal came back on for Black Friday. I brought it knowing that it would be on sale for 11/11 and the vendor had a 30 day price match policy after purchase. So I got it early and got the sale price.

I also have the LKS MH-DA004 on sale.

And not sure what you meant I figured out all the answers already. These are honest questions and up for members here to discuss.

View attachment 172710

You mentioned all the answers in the post already, at least in my point view.

It’s basically all about money and bragging rights, but at least on this forum we brag about real feats of engineering instead of bragging about who has the product with the best marketing material.

 

[gallantus]

You mentioned all the answers in the post already, at least in my point view.

It’s basically all about money and bragging rights, but at least on this forum we brag about real feats of engineering instead of bragging about who has the product with the best marketing material.

Honest answer. Love it!

Do you think if two amps that measures exactly alike but from different manufacturers with different design will sound exactly alike?

 

[restorer-john]

Do you think if two amps that measures exactly alike but from different manufacturers with different design will sound exactly alike?

You won't find two amplifiers that measure exactly the same from two different manufacturers, often not even the same model, from the same manufacturer...