Master Complaint Thread About Headphone Measurements

[Helicopter]

I guess that is where I find this headphone part of the site somewhat contradictory. On one hand, it is trying to empirically determine what is the best headphone based on a single target, going so far as to EQ every headphone with the aim to make everything match that single target, which by definition contradicts that everyone has their own ears and tastes. And on the other hand, it is mixing in that very same subjective ears and tastes in to the review, which we can all agree is unscientific. Perhaps a giant disclaimer at the top of the headphone section of this site would help others like me who was under the impression that the recommendations here were entirely based on science.

In blind testing, the minority of people who don't prefer something like the target curve tend to prefer something just like it, but with slightly more or less bass, so we know it is the best available approximation of the preferences of the vast majority of people, not just what sounds best to Amir's ears.

I wouldn't jump to the conclusion the listening tests re 100% subjective and unscientific. Amir has used the best scientific research available to make the listening tests much more objective and science-based than many reviewers. Obviously, comments like 'not putting a smile on his face' are subjective, but I we should be able to see that without a disclaimer. He's been quite transparent about his process.

 

[Jimbob54]

I guess that is where I find this headphone part of the site somewhat contradictory. On one hand, it is trying to empirically determine what is the best headphone based on a single target, going so far as to EQ every headphone with the aim to make everything match that single target, which by definition contradicts that everyone has their own ears and tastes. And on the other hand, it is mixing in that very same subjective ears and tastes in to the review, which we can all agree is unscientific. Perhaps a giant disclaimer at the top of the headphone section of this site would help others like me who was under the impression that the recommendations here were entirely based on science.

1. You have read the reviews? How can anyone conclude that the recommendation is purely science based when there is obviously a subjective element?

2. I think you misunderstand the purpose of the target and indeed its impact on the review /recommendation. NOTHING has hit the target fully. Nor do I think anything will.

 

[solderdude]

Perhaps a giant disclaimer at the top of the headphone section of this site would help others like me who was under the impression that the recommendations here were entirely based on science.

In every headphone review Amir has such a disclaimer:

Note: The measurements you are about to see are made using a standardized Gras 45C. Headphone measurements by definition are approximate and variable so don't be surprised if other measurements even if performed with the same fixtures as mine differ in end results. Protocols vary such as headband pressure and averaging (which I don't do). As you will see, I have confirmed the approximate accuracy of the measurements using Equalization and listening tests. Ultimately headphone measurements are less exact than speakers mostly in bass and above a few kilohertz so keep that in mind as you read these tests.

or

Note: The measurements you are about to see are preformed using standardized GRAS 45CA headphone measurement fixture. Headphone measurements require more interpretation than speaker tests and have more of a requirement for subjective testing as a result. In addition, comparison of measurements between different people performing it using different configurations requires fair bit of skill. So don't look for matching results. Focus on high level picture. Listening tests are performed using RME ADI-2 DAC and its headphone output.

It is just a matter of reading the whole review.

Rtings tried to do a 'science only' approach and got a lot of FLAK because the ratings didn't fit with people opinions.

There is nothing as subjective/personal in audio as headphones. You cannot look at a few graphs and know all there is to know about headphones.

 

[ethanchiu10]

In blind testing, the minority of people who don't prefer something like the target curve tend to prefer something just like it, but with slightly more or less bass, so we know it is the best available approximation of the preferences of the vast majority of people, not just what sounds best to Amir's ears.

I wouldn't jump to the conclusion the listening tests re 100% subjective and unscientific. Amir has used the best scientific research available to make the listening tests much more objective and science-based than many reviewers. Obviously, comments like 'not putting a smile on his face' are subjective, but I we should be able to see that without a disclaimer. He's been quite transparent about his process.

Well I feel like we going in circles here. If you say the target curve is such a good approximation that it hits the vast majority of people (by vast majority, I think you mean more than 67%, or 2/3rd of people ), then why don't we just stick to that, and we just give a recommendation based on how close we get to it? Or if we admit our data is only 67% good, then we have to admit our target curve is not really that great of a target, even though it may be the best we have for now. But I would hardly consider 67% accuracy to be "scientific".

Just adding my original comment here so other's can see where I'm coming from:

https://www.audiosciencereview.com/...l-celestee-review-headphone.21917/post-816020

I just spent several hours reading through the Focal Clear thread, and there you are always arguing about being scientific and about the importance of measurements over subjective feelings. But then here in this review, you conclude by saying even though the measurements are great, you personally don't "smile" when listening to this, and so you don't recommend this. So which is it, do you want to give recommendations strictly based on science? Or will you be giving your recommendations based on your subjective feelings? I feel like this review goes against everything you are trying to do here.

Using audiophile terms, I think that review is adding a whole lot of "noise" and "distortion" to the overall goal of this site. Is the goal here to be objective and scientific? Or is it just another subjective review site, just with some graphs that get ignored in the conclusion?

 

[ethanchiu10]

1. You have read the reviews? How can anyone conclude that the recommendation is purely science based when there is obviously a subjective element?

2. I think you misunderstand the purpose of the target and indeed its impact on the review /recommendation. NOTHING has hit the target fully. Nor do I think anything will.

Because it has been stated again and again that this site is called "Audio SCIENCE Review", with an emphasis on science. And I don't think I need to quote the numerous posts here where we are clearly aiming to be scientific and objective in our results here. Especially if you take a look at other categories of this site.

I don't think it is that unreasonable for anyone to conclude that the goal of this site was to be purely scientific. Let me put it this way. if we had the tools to be 100% accurate in our scientific assessment of headphones here, are you saying Amir would not use it? And he would still stick to his subjective opinion? No, I think its pretty clear the aim of this site is to be as close to scientifically accurate as we can possibly be using the latest research.

 

[Jimbob54]

Because it has been stated again and again that this site is called "Audio SCIENCE Review", with an emphasis on science. And I don't think I need to quote the numerous posts here where we are clearly aiming to be scientific and objective in our results here. Especially if you take a look at other categories of this site.

I don't think it is that unreasonable for anyone to conclude that the goal of this site was to be purely scientific. Let me put it this way. if we had the tools to be 100% accurate in our scientific assessment of headphones here, are you saying Amir would not use it? And he would still stick to his subjective opinion? No, I think its pretty clear the aim of this site is to be as close to scientifically accurate as we can possibly be using the latest research.

Yes, the reviews and amir's approach to HP is led by science but no, the science (or my limited understanding of it) is that we are a way off understanding and measuring fully all the characteristics of how HP sound. The science is far clearer for the electronics, hence less subjective elements to those reviews.

So what would you prefer, no HP reviews here? Or simply FR /distortion charts and the rest with zero commentary? I'm not sure either option better serves the membership or the industry than the current state.

 

[ethanchiu10]

In every headphone review Amir has such a disclaimer:



or



It is just a matter of reading the whole review.

Rtings tried to do a 'science only' approach and got a lot of FLAK because the ratings didn't fit with people opinions.

There is nothing as subjective/personal in audio as headphones. You cannot look at a few graphs and know all there is to know about headphones.

In my opinion, those quotes are not direct enough. When I read those comments, I thought, ok so we can trust the measurements 90%, and there is a tiny bit of 10% leeway for subjectivity.

But when you have a review like the Focal Celestee that measures so well, yet gets a not recommended, it feels like subjectivity is actually at least 51%, and measurements are only 49% of the result, and it makes me feel cheated as a reader of the site.

I just think a clearer direction would be helpful to everyone.

 

[ethanchiu10]

Yes, the reviews and amir's approach to HP is led by science but no, the science (or my limited understanding of it) is that we are a way off understanding and measuring fully all the characteristics of how HP sound. The science is far clearer for the electronics, hence less subjective elements to those reviews.

So what would you prefer, no HP reviews here? Or simply FR /distortion charts and the rest with zero commentary? I'm not sure either option better serves the membership or the industry than the current state.

I could think of several ways to go about it.

1. Yes, it could be a place for just very accurate charts, with no "recommended" or not labels.

2. The recommendation is based purely on the numbers (how much deviation from the best scientific target we know of)

3. It could be that there is 2 separate ratings. The best scientific rating we have, and Amir's own personal rating.

- Scientific Recommendation
- Scientific Recommendation with EQ
- Amir's Recommendation
- Amir's Recommendation with EQ

4. It can be an off-shoot site for Amir's personal opinions. (Imagine if this site was to become the de-factor scientific audio equipment site, and Amir is no longer the only person reviewing, it would get jumbled up more, a separate site for his own opinion would then make sense).

5. A clear disclaimer that this headphone section of the site is not scientific.

 

[solderdude]

In my opinion, those quotes are not direct enough. When I read those comments, I thought, ok so we can trust the measurements 90%, and there is a tiny bit of 10% leeway for subjectivity.

But when you have a review like the Focal Celestee that measures so well, yet gets a not recommended, it feels like subjectivity is actually at least 51%, and measurements are only 49% of the result, and it makes me feel cheated as a reader of the site.

I just think a clearer direction would be helpful to everyone.

The 90/10 instead of 49/51 is entirely your subjective opinion I would say.
There are no percentages. There are certain measurements done in certain conditions.
There is Amir's subjective evaluation.
There is Amir's confirmation on his vision of accurate by applying EQ based on findings.

There simply are no 'accurate' measurements for headphones. NOT a single one of them is accurate. The measurements at ASR are done on a 'standard' rig with a target chosen by Amir based on what he found was the most correct one.

It's not like the measurements are +/ -0.5dB accurate or so. Your assumption that headphone evaluation here is done only or mostly on science and a number put on it is probably caused by not reading or fully understanding what the measurements actually say despite the disclaimers in the articles.
That's hardly Amir's problem. Of course you are not the only one and suspect that the majority of readers will have similar thoughts but perhaps with different 'percentages' of science/opinion in mind.

What exact disclaimer would you feel would be needed to 'warn' potential readers about the science (more like engineering) part and what subjective findings of Amir would you think is needed for someone stumbling on the site and not read up on ASR.

 

[Zensō]

I could think of several ways to go about it.

1. Yes, it could be a place for just very accurate charts, with no "recommended" or not labels.

2. The recommendation is based purely on the numbers (how much deviation from the best scientific target we know of)

3. It could be that there is 2 separate ratings. The best scientific rating we have, and Amir's own personal rating.

- Scientific Recommendation
- Scientific Recommendation with EQ
- Amir's Recommendation
- Amir's Recommendation with EQ

4. It can be an off-shoot site for Amir's personal opinions. (Imagine if this site was to become the de-factor scientific audio equipment site, and Amir is no longer the only person reviewing, it would get jumbled up more, a separate site for his own opinion would then make sense).

5. A clear disclaimer that this headphone section of the site is not scientific.

Amir’s current disclaimers seem perfectly clear to me. In a nutshell, headphone testing is fraught with difficulties, so don’t focus too much on the fine details.

I’d suggest a read through of the headphone reviews on this site starting from the first and going forward. The methodology has improved significantly over time and at present represents what I think are the most accurate headphone reviews available (along with @solderdude ’s).

 

[ethanchiu10]

The 90/10 instead of 49/51 is entirely your subjective opinion I would say.
There are no percentages. There are certain measurements done in certain conditions.
There is Amir's subjective evaluation.
There is Amir's confirmation on his vision of accurate by applying EQ based on findings.

There simply are no 'accurate' measurements for headphones. NOT a single one of them is accurate. The measurements at ASR are done on a 'standard' rig with a target chosen by Amir based on what he found was the most correct one.

It's not like the measurements are +/ -0.5dB accurate or so. Your assumption that headphone evaluation here is done only or mostly on science and a number put on it is probably caused by not reading or fully understanding what the measurements actually say despite the disclaimers in the articles.
That's hardly Amir's problem. Of course you are not the only one and suspect that the majority of readers will have similar thoughts but perhaps with different 'percentages' of science/opinion in mind.

What exact disclaimer would you feel would be needed to 'warn' potential readers about the science (more like engineering) part and what subjective findings of Amir would you think is needed for someone stumbling on the site and not read up on ASR.

There are no exact percentages of course. But you cannot say that Amir has represented his reviews as more subjective than objective. This entire site has clearly represented itself to be much more scientific than subjective. So in terms of percentages, I wouldnt say I was far off.

I think most new readers of this site will have my same assumption. The way this site is presented is clearly about science rather than opinion. So when we have a section of the site that I’m now told cannot be done scientifically. I think a big clear disclaimer is warranted.

I have already listed some potential solutions on top. If you want a specific example of a disclaimer. It could be an alert box at the very top, with something short and concise like:

The science on headphones is not accurate, the recommendation in this review is largely based on Amir's personal opinion.

One day, Amir will want to retire, and it would be nice if this site can continue on without him as (I assume) he intended it, a place to cut through the snake oil, and find scientific objective reviews on audio equipment. This is in everyone's interests, the readers and consumers, the competitive market, and Amir himself too, because if this site can truly become what it is trying to be, he could sell it off for a nice chunk of retirement money to buy himself some audio toys

 

[ethanchiu10]

Amir’s current disclaimers seem perfectly clear to me. In a nutshell, headphone testing is fraught with difficulties, so don’t focus too much on the fine details.

I’d suggest a read through of the headphone reviews on this site starting from the first and going forward. The methodology has improved significantly over time and at present represents what I think are the most accurate headphone reviews available (along with @solderdude ’s).

If you think the current disclaimer perfectly clarifies that the headphone reviews are a completely different breed of reviews from the rest of the site, and that there is no way for any improvement to avoid any confusions, then I can only agree to disagree.

 

[solderdude]

I have already listed some potential solutions on top. If you want a specific example of a disclaimer. It could be an alert box at the very top, with something short and concise like:

The science on headphones is not accurate, the recommendation in this review is largely based on Amir's personal opinion.

The first part already is in his disclaimer but in other wording .

Amir reviews the headphone and measures them and also states his subjective findings as such.

It's Amirs review and his recommendation usually is accompanied by the how and why. I don't understand why people will have to be alerted with an 'alert' box saying the review is Amirs opinion when it is already written inside the review itself.

I can understand you are disappointed and expected something different but an alert box at the very top telling people the review is done by Amir and the conclusion thus is merely a personal opinion others may not agree with is a bit weird.

One day, Amir will want to retire, and it would be nice if this site can continue on without him as (I assume) he intended it, a place to cut through the snake oil, and find scientific objective reviews on audio equipment. This is in everyone's interests, the readers and consumers, the competitive market, and Amir himself too, because if this site can truly become what it is trying to be, he could sell it off for a nice chunk of retirement money to buy himself some audio toys

Tyll also thought his website would continue when he retired.
NwAvGuy also came and went and so was Golden Ears.
Nothing lasts forever and stays the same.

 

[ethanchiu10]

The first part already is in his disclaimer but in other wording .

Amir reviews the headphone and measures them and also states his subjective findings as such.

It's Amirs review and his recommendation usually is accompanied by the how and why. I don't understand why people will have to be alerted with an 'alert' box saying the review is Amirs opinion when it is already written inside the review itself.

I can understand you are disappointed and expected something different but an alert box at the very top telling people the review is done by Amir and the conclusion thus is merely a personal opinion others may not agree with is a bit weird.

I guess that's where I thought this site was different. I thought this was suppose to be a place of science rather than opinion. But maybe I'm the only one who misunderstood it?

Tyll also thought his website would continue when he retired.
NwAvGuy also came and went and so was Golden Ears.
Nothing lasts forever and stays the same.

That's kinda my point haha. It's up to Amir. But I think if he wants this site to continue, then it cannot be "his" review. It needs to be a scientific review.

 

[Jimbob54]

I guess that's where I thought this site was different. I thought this was suppose to be a place of science rather than opinion. But maybe I'm the only one who misunderstood it?



That's kinda my point haha. It's up to Amir. But I think if he wants this site to continue, then it cannot be "his" review. It needs to be a scientific review.

Some might think you just didn't like the Celeste review. I think it perfectly encapsulates the problems with headphones. But better this than reviewers heaping praise on shiny, heavy overpriced engineering disasters, no?

 

[ethanchiu10]

Some might think you just didn't like the Celeste review. I think it perfectly encapsulates the problems with headphones. But better this than reviewers heaping praise on shiny, heavy overpriced engineering disasters, no?

I don't care about the Celestee product if that is that you mean. What I care about is science. And science includes principles such as objectivity, reproducibility, evidence, causation. If you have a review where all the measurements are great, and it concludes with a 'not recommended', it goes against the scientific nature of this site, and to me as a reader, it confuses me on the intended goal of the site, and it makes me feel cheated as the 'investment' I put in reading all the reviews now seems like it no longer has the meaning that I thought it did.

 

[Jimbob54]

I don't care about the Celestee product if that is that you mean. What I care about is science. And science includes principles such as objectivity, reproducibility, evidence, causation. If you have a review where all the measurements are great, and it concludes with a 'not recommended', it goes against the scientific nature of this site, and to me as a reader, it confuses me on the intended goal of the site, and it makes me feel cheated as the 'investment' I put in reading all the reviews now seems like it no longer has the meaning that I thought it did.

Probably not the place for you then. Ask for your money back....

 

[ethanchiu10]

Why isn't it a place for me? I think Amir's goals very much aligns with what I want to read. I'm very much interested in objective scientific reviews of audio products. The only issue I have with it is that the conclusions in headphone reviews throws everything before it out the window.

 

[Jimbob54]

Why isn't it a place for me? I think Amir's goals very much aligns with what I want to read. I'm very much interested in objective scientific reviews of audio products. The only issue I have with it is that the conclusions in headphone reviews throws everything before it out the window.

Don't read the listening section or the conclusion then and be happy with the charts

 

[ethanchiu10]

Sure I could be happy with that. I'm just giving suggestions to improve this place. But it is up to Amir whether or not he feels my suggestions align with his vision.