• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Master Thread: Are measurements Everything or Nothing?

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
Seems like so many review threads get challenged with:

1. Measurements are not everything.

2. You all never listen.

3. I trust my ears, not graphs.

4. I don't listen to graphs. I listen to music.

5. You all must not listen to music at all.

6. Why don't you all buy the best SINAD gear?

7. I have heard your best SINAD gear and they sound terrible. I don't like any of this Chinese stuff.

8. You don't trust your ears. I/we do.

9. All these reviewers/youtubers/audophiles say these amps, DACs, etc. sound different and you say they don't. They can't all be wrong.

10. Surely designers have created certain house sound for each equipment which your measurements don't show.

11. Your measurements are only at one frequency. You need to also measure X, Y and Z like impulse response, slew rate, etc., etc.

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.

On and on...

I have had to answer these so many times that I thought it is time to stop having them go into every review as they are not product specific. From here on, any such questions should be posted here. Answers will be given in this thread and simply referenced in future challenges in other threads.

@AdamG247 and @BDWoody, please direct any future posts in review threads to here and not allow discussions there.

Thanks. You all are free to discuss this topic, provide answers, argue, whatever, in this thread. :)
 

_thelaughingman

Major Contributor
Forum Donor
Joined
Jan 1, 2020
Messages
1,324
Likes
1,943
I am going to hyperlink this thread to every response that happens to question validity of objectivity and measurements in any of the threads. Enough is enough as you said.
 

Digby

Major Contributor
Joined
Mar 12, 2021
Messages
1,632
Likes
1,555
@amirm I'd like to know your position on the question. I imagine you are somewhere in the middle or else why bother with subjective listening impressions in your reviews? In a number of these listening reviews you have said how you couldn't really hear a fault that you measured and that you weren't always sure whether EQ improved a dip/peak or not.

Sometimes you was surprised by how much you enjoyed a speaker, I think the JBL 4349 was one of these, even though the measurements were far from the best you had seen. It seems the speakers provided something in reproduction that the measurements themselves didn't make obvious, which would suggest that measurements (at least as they currently stand) are not the be all and end all, at least when it comes to enjoying music.

Your thoughts?
 
Last edited:

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,579
Likes
38,280
Location
Gold Coast, Queensland, Australia
@amirm This goes to the suggestion I made years ago.

Do a review thread, present it and lock it. Only make changes where there were errors or omissions.

Have a companion discussion thread with a link from the review thread. If it goes off the rails it can get moderated, locked, whatever and unlinked/deleted without poisoning the main review thread. The discussion threads will die down eventually and the review threads will still be clean, full of facts and a useful resource.

I don't agree with attempting to sanitise discussion on particular review products or your testing methodology by steering people to a catch all, out of the way thread. It isn't a good look at all IMO. And, it won't work anyway- people will still discuss whatever they want and you will only give the moderators a whole lot more work.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
@amirm I'd like to know your position on the question. I imagine you are somewhere in the middle or else why bother with subjective listening impressions in your reviews? In a number of these listening reviews you have said how you couldn't really hear a fault that you measured and that you weren't always sure whether EQ improved a dip/peak or not.

Sometimes you was surprised by how much you enjoyed a speaker, I think the JBL 4349 was one of these, even though the measurements were far from the best you had seen. It seems the speakers provided something in reproduction that the measurements themselves didn't make obvious, which would suggest that measurements (at least as they currently stand) are not the be all and end all, at least when it comes to enjoying music.

Your thoughts?
I have a scale for how much measurements matter for each category of products:

DACs: 100%
Amplifiers (headphone and speaker): 80 to 90% due to variability of available power. Hard to internalize how much power is available/enough without listening tests.
Speakers: 70 to 80%
Headphones: 50 to 80% (measurements too variable)

This is why you see me do listening tests for the last two categories and half of second (headphone amps).
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
@amirm This goes to the suggestion I made years ago.

Do a review thread, present it and lock it. Only make changes where there were errors or omissions.
Member questions/comments help make the reviews better. Putting them elsewhere makes it hard for members to reference what is in the original review.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
I don't agree with attempting to sanitise discussion on particular review products or your testing methodology by steering people to a catch all, out of the way thread.
If the comment is specific to the product being reviewed, then it is fine to be asked and discussed there. This thread is for the generic comments that apply to any review we do here.
 

_thelaughingman

Major Contributor
Forum Donor
Joined
Jan 1, 2020
Messages
1,324
Likes
1,943
Can we expand on this further and make this a sticky somewhere.

Guidelines to engage in meaningful conversations on ASR

- Engage in a topic from the perspective of wanting to learn something new about the hobby of being an audiophile.
- Contribute to the topic/thread with good knowledge that has some semblance of support in objective measurements or proven science.
- Disengage from making statements or claims that are steeped in subjective opinion. ie: a transparent dac at $5k is better sounding than a dac at $150. Etc
 

gallantus

Active Member
Joined
May 7, 2021
Messages
135
Likes
144
Alright, I will kick this off!!! Thanks @amirm for setting this up.

First, I believe in measurements. Every gear I buy I need to look at measurements if available. I own the Topping D90SE, because it measures the best, the reason why I brought it, amongst the functionality and the fact that it was on sale. However, I didn't keep it because it measures the best, because I thought it sounds terrific. I am a trained EE, albeit non practicing. As an engineer, I say, you need to measure it, in order to build it.

Also let me get out of the way, there are legit snake oil out there and you know what I am talking about.

But is measurement for electronics the end all be all to guage sound quality?

If two amps measured exactly alike, but they are from different manufacturers with different design, will they sound exactly alike?

Can one buy an amp or DAC, just by looking at the measurements without any audition and know that it will sound like any other amp or DAC only less or more distortion?

If so, why are there so many people not have the Benchmark AHB2 and LA4 and the Topping D90SE combination (budget and use case functionality aside). After all, they measured the best. Yes, yes, some will say because, I need more power (you can bridge the Benchmark into mono btw), some will say I need some other functionalities, etc etc.

Why does x SINAD, y THD and z SNR matter when your ears likely can't tell the difference once you hit a certain threshold? Is it for bragging rights? A peace of mind?

These are honest questions, maybe a bit exoteric and philosophical, but honest questions nevertheless.
 

gallantus

Active Member
Joined
May 7, 2021
Messages
135
Likes
144
I have a scale for how much measurements matter for each category of products:

DACs: 100%
Amplifiers (headphone and speaker): 80 to 90% due to variability of available power. Hard to internalize how much power is available/enough without listening tests.
Speakers: 70 to 80%
Headphones: 50 to 80% (measurements too variable)

This is why you see me do listening tests for the last two categories and half of second (headphone amps).
This is super helpful, I don't think I have seen your position on this anywhere (or at least my first time seeing it). Your position answered a lot of my questions :D
 

BoredErica

Addicted to Fun and Learning
Joined
Jan 15, 2019
Messages
629
Likes
899
Location
USA
2 cents... Some would say they're worth even less than that but here it goes... :)

5. You all must not listen to music at all.
Subjectivists bring this up to take a jab at objectivists but it's so absurd I don't know how to respond to it. What do subjectivists think objectivists actually do after they make fancy graphs and spend a ton of money on their system? They just let the gear sit there? Make endless graphs forever? I listen to my speakers every day! Gotta get my money's worth.

3. I trust my ears, not graphs.
Human brains are incredibly fallible. Placebo is real and that's why medical trials control for it. I'm glad when I board an airplane the airplane wasn't built based off of intuition alone, or the drugs I take tested by giving them to the scientists friends and family with their effectiveness judged based on their feedback. People want to believe they have an entire hobby going on here where everything sounds different. It's cool and people have bought into it because a bunch of people who have no business teaching others about audio said some amps are more 'musical' than other amps or the Skin Effect ruins audio cables. I'm more interested in how reality actually works rather than what makes me feel good.

Ironically it's the objectivists who trust their ears. I trust my ears, so I'd like only my ears tested whenever possible. I don't need my eyes to tell me what I'm listening to. It's the subjectivists who don't trust their ears. So many self proclaimed audio gurus all of a sudden fail to spot differences between two transparent pieces of gear when they can't see what they're listening to.

I don't listen to graphs. I listen to music.
And how that music sounds is based on real things that are described in graphs. :)

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.
I can't even use the word 'double blind testing' in most areas of Headfi and most of audio spaces online be it Youtube or forums are dominated by subjectivists, so it's no surprise the few spots for objectivists don't take too kindly to subjectivists when they cause others to waste their money on things that don't make a difference, encouraging others to peddle yet more snake oil to unsuspecting masses.

6. Why don't you all buy the best SINAD gear?
I don't have infinite money. I care about how products look and various features. SINAD is not the end all be all of audio measurements, otherwise Amir and others can save a whole lot of time ignoring every other test.

--
Even for people adamant that they like a certain colored sound, knowing what it is they like allows them to shop better or EQ whatever they like in better. Knowing how reality works makes things more complicated yes, but also empowers you to do things you otherwise would not have been able to.

Also for people who aren't quite as much of an objectivist as I am, there is room for grey area. You don't have to think 'all of my subjective feelings about this headphone are meaningless'. You could instead think 'well I know how headphones sound is not an entirely 100% settled on measurements on a standard head alone since it can vary based on individual things like head size and earcup seal, so my subjective thoughts might be meaningful, though I still take them with a grain of salt given what I know about how fallible human brains are.' Of course, it only makes sense to have a grey area filled in with subjective opinions on how things sound when it comes to areas where measurements alone are not enough. Where measurements are enough to describe how something sounds, subjective opinions are worthless.
 

abdo123

Master Contributor
Forum Donor
Joined
Nov 15, 2020
Messages
7,424
Likes
7,940
Location
Brussels, Belgium
Alright, I will kick this off!!! Thanks @amirm for setting this up.

First, I believe in measurements. Every gear I buy I need to look at measurements if available. I own the Topping D90SE, because it measures the best, the reason why I brought it, amongst the functionality and the fact that it was on sale. However, I didn't keep it because it measures the best, because I thought it sounds terrific. I am a trained EE, albeit non practicing. As an engineer, I say, you need to measure it, in order to build it.

Also let me get out of the way, there are legit snake oil out there and you know what I am talking about.

But is measurement for electronics the end all be all to guage sound quality?

If two amps measured exactly alike, but they are from different manufacturers with different design, will they sound exactly alike?

Can one buy an amp or DAC, just by looking at the measurements without any audition and know that it will sound like any other amp or DAC only less or more distortion?

If so, why are there so many people not have the Benchmark AHB2 and LA4 and the Topping D90SE combination (budget and use case functionality aside). After all, they measured the best. Yes, yes, some will say because, I need more power (you can bridge the Benchmark into mono btw), some will say I need some other functionalities, etc etc.

Why does x SINAD, y THD and z SNR matter when your ears likely can't tell the difference once you hit a certain threshold? Is it for bragging rights? A peace of mind?

These are honest questions, maybe a bit exoteric and philosophical, but honest questions nevertheless.

Seems like you figured out all the answers already.

the only thing surprising here is that you say you bought the D90SE on sale, which is very surprising tbh.

you should never do that to a flagship product, it brings its market value down. Now people will only buy it at its discounted price.
 

gallantus

Active Member
Joined
May 7, 2021
Messages
135
Likes
144
Seems like you figured out all the answers already.

the only thing surprising here is that you say you bought the D90SE on sale, which is very surprising tbh.

you should never do that to a flagship product, it brings its market value down. Now people will only buy it at its discounted price.

I be damn shocked if no one else saw/know about this deal. In fact, this deal came back on for Black Friday. I brought it knowing that it would be on sale for 11/11 and the vendor had a 30 day price match policy after purchase. So I got it early and got the sale price.

I also have the LKS MH-DA004 on sale.

And not sure what you meant I figured out all the answers already. These are honest questions and up for members here to discuss.

Screenshot_20211215-192437~2.png
 
Last edited:

abdo123

Master Contributor
Forum Donor
Joined
Nov 15, 2020
Messages
7,424
Likes
7,940
Location
Brussels, Belgium
I be damn shocked if no one else saw/know about this deal. In fact, this deal came back on for Black Friday. I brought it knowing that it would be on sale for 11/11 and the vendor had a 30 day price match policy after purchase. So I got it early and got the sale price.

I also have the LKS MH-DA004 on sale.

And not sure what you meant I figured out all the answers already. These are honest questions and up for members here to discuss.

View attachment 172710
You mentioned all the answers in the post already, at least in my point view.

It’s basically all about money and bragging rights, but at least on this forum we brag about real feats of engineering instead of bragging about who has the product with the best marketing material.
 

gallantus

Active Member
Joined
May 7, 2021
Messages
135
Likes
144
You mentioned all the answers in the post already, at least in my point view.

It’s basically all about money and bragging rights, but at least on this forum we brag about real feats of engineering instead of bragging about who has the product with the best marketing material.
Honest answer. Love it!

Do you think if two amps that measures exactly alike but from different manufacturers with different design will sound exactly alike?
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,579
Likes
38,280
Location
Gold Coast, Queensland, Australia
Do you think if two amps that measures exactly alike but from different manufacturers with different design will sound exactly alike?

You won't find two amplifiers that measure exactly the same from two different manufacturers, often not even the same model, from the same manufacturer...
 

gallantus

Active Member
Joined
May 7, 2021
Messages
135
Likes
144
You won't find two amplifiers that measure exactly the same from two different manufacturers, often not even the same model, from the same manufacturer...
True, but respectfully, that is a cop out answer.

So let me ask the question in another way. If two amps measured damn close from two different manufacturers with different design, will they sound damn close alike. Keep in mind, once you hit a certain SINAD, it makes no difference to our human ears anyway.

Is the concept of "sound signature" BS?
 
Top Bottom