Master Complaint Thread About Headphone Measurements

[ethanchiu10]

Let me put it another way. For example, in his review of the Audio Technica ATH-ADX5000, or infamously, the Abyss Diana, he gave them a "not recommended" because they had significant distortion. But we know from tons of people that have listened to it that they loved it. If Amir had also subjectively loved the way it sounded, and given them a "recommended", despite the huge measurement flaws, would that be acceptable here?

Like I said, theres just a lot of "noise" and "distortion" in here that I think could use a rethink.

 

[edahl]

Product reviews aren't science.

 

[solderdude]

Maybe it would be sufficient if Amir would put up a notification like:

Review also contains subjective findings due to the nature of the reviewed product.

or something in that vein ?

I guess that's where I thought this site was different. I thought this was suppose to be a place of science rather than opinion. But maybe I'm the only one who misunderstood it?

Nope.
The problem is that headphones by design cannot be fully characterized by measurements alone.
It is possible for most electronic components and that's what started ASR. Later followed by speakers and recently headphones.
The latter are far more complex than what they seem to be.

 

[Robbo99999]

I don't care about the Celestee product if that is that you mean. What I care about is science. And science includes principles such as objectivity, reproducibility, evidence, causation. If you have a review where all the measurements are great, and it concludes with a 'not recommended', it goes against the scientific nature of this site, and to me as a reader, it confuses me on the intended goal of the site, and it makes me feel cheated as the 'investment' I put in reading all the reviews now seems like it no longer has the meaning that I thought it did.

Don't worry about it so much, there's no exact formula for how Amir combines his subjective experience with the objective measurements to end up with a panther rating & recommendation. The objective measurements are there for you to see, and so is Amir's subjective assessment, it's up to you to decide how much weight to place on Amir or on the objective measurements. If you build up some experience EQ'ing different headphones and comparing them, then this helps you to put the objective measurements into perspective for yourself and would give you more confidence on which to make conclusions based on the objective measurements, as well as perspective on Amir's subjective review. For people with less experience I think it probably pays to put more weight on Amir's subjective impressions.

 

[Jimbob54]

Let me put it another way. For example, in his review of the Audio Technica ATH-ADX5000, or infamously, the Abyss Diana, he gave them a "not recommended" because they had significant distortion. But we know from tons of people that have listened to it that they loved it. If Amir had also subjectively loved the way it sounded, and given them a "recommended", despite the huge measurement flaws, would that be acceptable here?

Like I said, theres just a lot of "noise" and "distortion" in here that I think could use a rethink.

This is why some folks don't like the measure then listen approach. It colours the subjective opinion.

 

[ethanchiu10]

Well both the measure-first and listen-first approach are flawed, because they both conclude with a subjective opinion. The simple solution is to just conclude based on measured data. The same way it is done in other categories such as amps/dacs. And if our current level of headphone science is only 67% accurate, then state so clearly in the beginning that this section of the website is entirely different than the rest. Not simply saying "oh note that measurements are not that accurate." How not accurate? So inaccurate to be completely useless? So inaccurate that a headphone can have perfect measurements and sound so bad that they are "not recommended"?

Sure, I could be happy just looking at the graphs. Sure I could simply not worry, and just accept the site as it is and not comment. But I'm participating and commenting here because I want to help improve this site as a useful tool for everyone.

And I just want to add, I admire Amir's work here. This is a really great resource in a lot of ways. I just think this headphone section needs some rethinking on its approach.

 

[Jimbob54]

Well both the measure-first and listen-first approach are flawed, because they both conclude with a subjective opinion. The simple solution is to just conclude based on measured data. The same way it is done in other categories such as amps/dacs. And if our current level of headphone science is only 67% accurate, then state so clearly in the beginning that this section of the website is entirely different than the rest. Not simply saying "oh note that measurements are not that accurate." How not accurate? So inaccurate to be completely useless? So inaccurate that a headphone can have perfect measurements and sound so bad that they are "not recommended"?

Sure, I could be happy just looking at the graphs. Sure I could simply not worry, and just accept the site as it is and not comment. But I'm participating and commenting here because I want to help improve this site as a useful tool for everyone.

And I just want to add, I admire Amir's work here. This is a really great resource in a lot of ways. I just think this headphone section needs some rethinking on its approach.

I feel like you are hell bent on doing a lot of talking and hand waving and not a whole lot of listening/ reading.

Also, you misunderstand how the site is structured. There is no real place to put such a disclaimer apart from in each review. As is done already.

 

[ethanchiu10]

I've done a lot of listening and reading, I just don't accept the answers as being good enough. I'm not the one hand waving away a very clear hypocrisy here. Let me write my complain more concisely:

1. There are reviews that measure poorly, and are guaranteed to be given a poor rating regardless of how it sounds. Because measurements and science matters here ofcource. e.g. Abyss Diana

2. There are reviews with very good measurements in every way (hits the harman target, low distortion, etc.), but they apparently still sound so bad that they are not recommended. Because measurements are too inaccurate / science don't matter. e.g. Focal Celestee

To me, I would like to see Amir make up his mind on which direction to take this site. But maybe to a lot of you, it doesn't matter, you don't really care about science or consistency. You are just here for Amir's personal opinion. That's fine, but it matters to me, and that's what I'm here to talk about. I thought this place was for people that cared about science, but it looks like I was wrong.

 

[Robbo99999]

Well both the measure-first and listen-first approach are flawed, because they both conclude with a subjective opinion. The simple solution is to just conclude based on measured data. The same way it is done in other categories such as amps/dacs. And if our current level of headphone science is only 67% accurate, then state so clearly in the beginning that this section of the website is entirely different than the rest. Not simply saying "oh note that measurements are not that accurate." How not accurate? So inaccurate to be completely useless? So inaccurate that a headphone can have perfect measurements and sound so bad that they are "not recommended"?

Sure, I could be happy just looking at the graphs. Sure I could simply not worry, and just accept the site as it is and not comment. But I'm participating and commenting here because I want to help improve this site as a useful tool for everyone.

And I just want to add, I admire Amir's work here. This is a really great resource in a lot of ways. I just think this headphone section needs some rethinking on its approach.

Headphones & speakers have to be listened to in order to draw a conclusion on them for the review, so absolutely Amir needs to listen to the speakers & headphones he's testing, he's also gotta relay his listening experience.......he does all this already. It's more important for headphones due the measurements not characterising the experience as well as that of speakers. For instance, from my own perspective, perhaps the most major element I'm interested in from Amir's subjective listening experience is his experience of soundstage of a headphone, as this is an important part of the headphone experience (& certainly for me) yet there are no measurements that characterise the soundstage experience......so his subjective impression on this matter is the only way to get at that important aspect of headphone performance. I feel confident enough to guage the performance of the rest of the headphone from his measurements, apart from his notes on headphone fitting & build quality & unique experiences like noticing the earcup resonating when it shouldn't or noticing the driver clipping when it shouldn't for instance.

I've done a lot of listening and reading, I just don't accept the answers as being good enough. I'm not the one hand waving away a very clear hypocrisy here. Let me write my complain more concisely:

1. There are reviews that measure poorly, and are guaranteed to be given a poor rating regardless of how it sounds. Because measurements and science matters here ofcource. e.g. Abyss Diana

2. There are reviews with very good measurements in every way (hits the harman target, low distortion, etc.), but they apparently still sound so bad that they are not recommended. Because measurements are too inaccurate / science don't matter. e.g. Focal Celestee

To me, I would like to see Amir make up his mind on which direction to take this site. But maybe to a lot of you, it doesn't matter, you don't really care about science or consistency. You are just here for Amir's personal opinion. That's fine, but it matters to me, and that's what I'm here to talk about. I thought this place was for people that cared about science, but it looks like I was wrong.

Regarding the Abyss Diana, I think it's important that some kind of engineering standards are met re various variables like distortion when it comes to uber expensive headphones like this one, and that was addressed in the review, and besides he found that headphone unexciting even after EQ....so it didn't really meet any of the standards be they subjective or objective.

About the Focal Celestee, to be honest it didn't measure well either, the frequency response was very jagged above 2kHz which would be unfixable in terms of removing all that jaggedness and there was quite large channel imbalance in the bass. Bass distortion wasn't particularly good in that headphone, although it was ok. Amir's subjective review was also not liking the headphone with poor soundstage and unimpressive bass even after EQ.....so really for me that headphone didn't measure well & also did not get a good subjective review, so I have a different take on that one to you - there's no conflict between the objective & the subjective in the review for this headphone from my perspective.

If you don't have much experience then definitely put lots of weight on all of Amir's subjective reviews, but if you have a fair amount of headphone EQ experience/experimentation experience between a few different headphones then I think it's safe to place more weight on the objective measurements that you see, but you still have to take note of Amir's notes on soundstage and other unmeasurable items like build quality / fitment / strange stuff like resonances or driver clipping.

 

[aandres_gm]

I honestly think when it comes to speakers and headphones, personal taste matters too much. It's why I don't read much into measurements (outside of EQ purposes) and when it comes to the subjective thoughts a reviewer may have, I tend to stick around mostly for the physical aspects. When it comes to the sound, I try to stick to reviewers who have in past occasions favoured headphones with sound signatures I live, but even then, I take everything with a grain of salt.

Something I'd really like, however, is for this site to be objective with the measurements. The review on the Hifiman Ananda was a total joke, in my opinion: the headphone wasn't fitted properly, as noted by several other reviewers, and this clearly tainted the subjective part of the review. Worst part was how the tester completely refused to re-measure with proper fitting and even went the lengths of claiming the test was accurate, as that's the kind of fit he'd expect customers to have anyway (a claim that wasn't be proven, which is pretty much this forum's reason of being).

99% of the time I simply ignore both this and the speaker review sections.

 

[Jimbob54]

I've done a lot of listening and reading, I just don't accept the answers as being good enough. I'm not the one hand waving away a very clear hypocrisy here. Let me write my complain more concisely:

1. There are reviews that measure poorly, and are guaranteed to be given a poor rating regardless of how it sounds. Because measurements and science matters here ofcource. e.g. Abyss Diana

2. There are reviews with very good measurements in every way (hits the harman target, low distortion, etc.), but they apparently still sound so bad that they are not recommended. Because measurements are too inaccurate / science don't matter. e.g. Focal Celestee

To me, I would like to see Amir make up his mind on which direction to take this site. But maybe to a lot of you, it doesn't matter, you don't really care about science or consistency. You are just here for Amir's personal opinion. That's fine, but it matters to me, and that's what I'm here to talk about. I thought this place was for people that cared about science, but it looks like I was wrong.

He's very clearly made up his mind how he wants to do headphone reviews. This clearly causes you some dissonance. He has had this debate many times on here, in this thread and the review threads. Lets not go down the "you dont agree with me so you don't care about science/ measurements/ consistency" route.

Read his review up to the listening/ conclusion and be happy.

EDIT- and if you do, you'll end up with (what Amir considers to be ) a boring headphone. YMMV.

 

[edahl]

I think it's a very clear error to mistake reviews informed by science with reviews being scientific *in themselves*.

 

[solderdude]

I've done a lot of listening and reading, I just don't accept the answers as being good enough. I'm not the one hand waving away a very clear hypocrisy here. Let me write my complain more concisely:

1. There are reviews that measure poorly, and are guaranteed to be given a poor rating regardless of how it sounds. Because measurements and science matters here ofcource. e.g. Abyss Diana

It looks like you don't agree there is poor technical performance here or do you suggest it doesn't matter as long as it sounds good.
Since you seem to be science oriented why would you disagree with the findings ?

2. There are reviews with very good measurements in every way (hits the harman target, low distortion, etc.), but they apparently still sound so bad that they are not recommended. Because measurements are too inaccurate / science don't matter. e.g. Focal Celestee

Not because measurements are too inaccurate / science don't matter. Amir found some others issues and was disappointed (happens sometimes)

To me, I would like to see Amir make up his mind on which direction to take this site. But maybe to a lot of you, it doesn't matter, you don't really care about science or consistency. You are just here for Amir's personal opinion. That's fine, but it matters to me, and that's what I'm here to talk about. I thought this place was for people that cared about science, but it looks like I was wrong.

This is Amir's forum. It's his and he can do with it what he wants and how he wants to.
When you enter someone's house are you going to stomp around demanding they change the furniture to your liking ?

What I found over the years is that if you want Amir to think about something or change things you can suggest things and give some good reasons as to why. When he doesn't agree with your ideas that's it. No need to whine about it.
When you approach someone (thus also Amir) demanding things he simply will not even listen to your 'requests'.

Slow down, make a clear case, back it up with science if you can. Barging in and expecting someone to change things so it suits you better is well... erm... it wouldn't work with me either.

You're in Amir's 'house' not in an ASR democracy where ASR is changed so that others feel it is more to their liking.

 

[Zensō]

If you think the current disclaimer perfectly clarifies that the headphone reviews are a completely different breed of reviews from the rest of the site, and that there is no way for any improvement to avoid any confusions, then I can only agree to disagree.

I don’t see what the confusion is. Headphones are difficult to test at the same level of accuracy as speakers, much less electronics. Amir mitigates for this by EQing to a consistent target and then testing his EQ adjustments through listening, which either confirms or calls into question the measurements. I see of no other way to check the validity of the measurements. Do you have a better method in mind? If so, I’m sure we’d all like to hear it.

 

[Jimbob54]

I don’t see what the confusion is. Headphones are difficult to test at the same level of accuracy as speakers, much less electronics. Amir mitigates for this by EQing to a consistent target and then testing his EQ adjustments through listening, which either confirms or calls into question the measurements. I see of no other way to check the validity of the measurements. Do you have a better method in mind? If so, I’m sure we’d all like to hear it.

Also cross references to others on similar rigs. Apparently.

 

[ethanchiu10]

Headphones & speakers have to be listened to in order to draw a conclusion on them for the review, so absolutely Amir needs to listen to the speakers & headphones he's testing, he's also gotta relay his listening experience.......he does all this already. It's more important for headphones due the measurements not characterising the experience as well as that of speakers. For instance, from my own perspective, perhaps the most major element I'm interested in from Amir's subjective listening experience is his experience of soundstage of a headphone, as this is an important part of the headphone experience (& certainly for me) yet there are no measurements that characterise the soundstage experience......so his subjective impression on this matter is the only way to get at that important aspect of headphone performance. I feel confident enough to guage the performance of the rest of the headphone from his measurements, apart from his notes on headphone fitting & build quality & unique experiences like noticing the earcup resonating when it shouldn't or noticing the driver clipping when it shouldn't for instance.

About the Focal Celestee, to be honest it didn't measure well either, the frequency response was very jagged above 2kHz which would be unfixable in terms of removing all that jaggedness and there was quite large channel imbalance in the bass. Bass distortion wasn't particularly good in that headphone, although it was ok. Amir's subjective review was also not liking the headphone with poor soundstage and unimpressive bass even after EQ.....so really for me that headphone didn't measure well & also did not get a good subjective review, so I have a different take on that one to you - there's no conflict between the objective & the subjective in the review for this headphone from my perspective.

It looks like you don't agree there is poor technical performance here or do you suggest it doesn't matter as long as it sounds good.
Since you seem to be science oriented why would you disagree with the findings ?

Now this I can accept, seems my initial interpretation of the review was wrong, and I had misread it, and the measurements are in fact poor? If that is the case, then it is good that it got not recommended and I have no issue there.

 

[ethanchiu10]

He's very clearly made up his mind how he wants to do headphone reviews. This clearly causes you some dissonance. He has had this debate many times on here, in this thread and the review threads. Lets not go down the "you dont agree with me so you don't care about science/ measurements/ consistency" route.

Read his review up to the listening/ conclusion and be happy.

EDIT- and if you do, you'll end up with (what Amir considers to be ) a boring headphone. YMMV.

Wow, thats actually exactly what you are doing. You don't agree with me, and you are hand waving me away. I stated my case clearly, and I wanted a real answer. Not this.

 

[ethanchiu10]

I think it's a very clear error to mistake reviews informed by science with reviews being scientific *in themselves*.

That is very well my mistake. Well put. In my defence, I think that was a fair mistake to make in a place called Audio Science Review. I would argue that the reviews in the other section are actually scientific "in themselves" too. A more accurate (but obviously unwieldy) name would be something like "Amir's headphone reviews informed by science" : P

 

[ethanchiu10]

This is Amir's forum. It's his and he can do with it what he wants and how he wants to.
When you enter someone's house are you going to stomp around demanding they change the furniture to your liking ?

What I found over the years is that if you want Amir to think about something or change things you can suggest things and give some good reasons as to why. When he doesn't agree with your ideas that's it. No need to whine about it.
When you approach someone (thus also Amir) demanding things he simply will not even listen to your 'requests'.

Slow down, make a clear case, back it up with science if you can. Barging in and expecting someone to change things so it suits you better is well... erm... it wouldn't work with me either.

You're in Amir's 'house' not in an ASR democracy where ASR is changed so that others feel it is more to their liking.

But we are not in Amir's house. This is not a private forum. It is actually a public building calling itself "Audio Science Review", and I think it is fair that it is subject to public scrutiny.

I have no interest in barging in to someone's house and expecting someone to change things to suit my needs. You have misinterpreted me. I interpret it as more like: Someone has made a post on a what appears to be a scientific journal (e.g. Nature journal), and I'm scrutinising whether or not it was done in a proper scientific method. That is how science has always been done.

Obviously, this is not a proper scientific journal, and maybe Amir's vision never meant it to be. At the end of the day, it is his project, and I've stated on multiple occasions that it is always up to him at the end. I'm just making suggestions.

 

[ethanchiu10]

I honestly think when it comes to speakers and headphones, personal taste matters too much. It's why I don't read much into measurements (outside of EQ purposes) and when it comes to the subjective thoughts a reviewer may have, I tend to stick around mostly for the physical aspects. When it comes to the sound, I try to stick to reviewers who have in past occasions favoured headphones with sound signatures I live, but even then, I take everything with a grain of salt.

Something I'd really like, however, is for this site to be objective with the measurements. The review on the Hifiman Ananda was a total joke, in my opinion: the headphone wasn't fitted properly, as noted by several other reviewers, and this clearly tainted the subjective part of the review. Worst part was how the tester completely refused to re-measure with proper fitting and even went the lengths of claiming the test was accurate, as that's the kind of fit he'd expect customers to have anyway (a claim that wasn't be proven, which is pretty much this forum's reason of being).

99% of the time I simply ignore both this and the speaker review sections.

Yeah, I was under the impression that this place contained scientific review. And it largely is. Except the headphone section.