Master Complaint Thread About Headphone Measurements

[SamV]

Being closer to average doesn't do you any good if you can't correlate that with listener preference. The goal here is not to just spit out some graphs. It is a means to an end of finding out if a headphone is good sounding or not. 45C is doing a superb job for me after testing a number of headphones. I see no reason for anything else and randomness that brings.

Hi!

As I've mentioned before elsewhere, after spending two weeks with the 5128 and measuring more than 50 headphones on it, I found it to be problematic with certain in-ears to the point that I would call it faulty. As you said, even if it is closer to average, it doesn't mean it's better. Certain IEMs would just fall out of its ear canal due to poor seal, and I couldn't replicate that on any human subject. As for its performance with over/on-ears go, it was better than the HMS and KEMAR, but not enough to justify its price, especially since the majority of improvement was above 9kHz. It didn't perform significantly better than the HMS in the bass range and showed some seal issues, so I believe measuring the bass on multiple humans is still the best way to go.

My current overall favorite system is the HMS (which is not without its issues), but if B&K fix the ear canal issue with the 5128, then it would be on top. I also spent two weeks with KEMAR and found it to be much worse than I was expecting! The 45CA actually measures better than KEMAR, because of its flat plates.

 

[edahl]

My main complaint is there isn't enough of them!

 

[SamV]

On another note, I might be a little late, but I read this somewhere here so thought to clarify it. The final target I developed at RTINGS (test bench v1.4) complies with the Harman target as much as technically possible. It was derived by applying the Harman's transfer function ([Harman Target] - [45CA flat in-room]) to the DF response of the HMS. Since we use different dummy heads, and that we had also previously shown the DF response of the HMS to be very similar to the in-room response of the 45CA, this is the closest we could get to the target.

And here is the frequency response of the Galaxy Buds+ which is tuned to the Harman target compensated with the RTINGS v1.4 target:

https://www.rtings.com/headphones/1-4/graph#1791/3992

 

[edahl]

On another note, I might be a little late, but I read this somewhere here so thought to clarify it. The final target I developed at RTINGS (test bench v1.4) complies with the Harman target as much as technically possible. It was derived by applying the Harman's transfer function ([Harman Target] - [45CA flat in-room]) to the DF response of the HMS. Since we use different dummy heads, and that we had also previously shown the DF response of the HMS to be very similar to the in-room response of the 45CA, this is the closest we could get to the target.

And here is the frequency response of the Galaxy Buds+ which is tuned to the Harman target compensated with the RTINGS v1.4 target:

https://www.rtings.com/headphones/1-4/graph#1791/3992

Would it make sense to compare a headphone to several different targets and classify them by best match?

 

[SamV]

Would it make sense to compare a headphone to several different targets and classify them by best match?

Not sure I'm following, could you elaborate?

 

[amirm]

Certain IEMs would just fall out of its ear canal due to poor seal, and I couldn't replicate that on any human subject.

Oh, I thought it was just me! I had the exact same experience. Ruined a bunch of tips on my Etymotic and none would stay in there.

 

[amirm]

@SamV, are you Sam from Rting? If so, warm welcome!

 

[SamV]

Oh, I thought it was just me! I had the exact same experience. Ruined a bunch of tips on my Etymotic and none would stay in there.

Haha, ruined our Etymotic tips too!

 

[SamV]

@SamV, are you Sam from Rting? If so, warm welcome!

I am Sam, formerly of RTINGS. Thanks

 

[edahl]

Not sure I'm following, could you elaborate?

I'm not read up enough on it to have much to contribute; I'm expecting to have misunderstood something. But presumably you're comparing FR to the Harman target and giving a score based on deviation/match. If so, would it make sense to simultaneously score against a diffuse field target, Harman 2013, 2018, whichever, and classify a headphone by best match rather than picking a preference curve?

 

[SamV]

I'm not read up enough on it to have much to contribute; I'm expecting to have misunderstood something. But presumably you're comparing FR to the Harman target and giving a score based on deviation/match. If so, would it make sense to simultaneously score against a diffuse field target, Harman 2013, 2018, whichever, and classify a headphone by best match rather than picking a preference curve?

Sure, studies have shown that the latest Harman target is preferred by the majority, but that doesn't mean everyone likes that target. We had discussions at RTINGS 2-3 years ago about supporting multiple targets or even allowing users to create their own target, but that would just add too much complexity to the website. I'm not working there anymore, so not sure what their current plans are. But the company is going through a period of expansion now, so any major changes to the review structure will probably happen only after that. I also believe they still prefer simplicity over complexity, so that feature may never appear.

The bigger issue in my opinion is that none of the current dummy heads are good enough. They have issues in both the bass and treble ranges. We tried to accomodate for the bass issues by measuring them on multiple humans, but the treble range issue is much harder to solve. For example, certain headphones show a noticeable dip in the treble range when compared to the target, but when you listen to it you can't really hear it. And when the dip is fixed with EQ, the headphone sounds too bright and much worse. Yet, there are other headphones that show a similar dip in the treble range, which you can hear and fix with EQ. I don't think the 5128 has solved that issue either. It may just be related to how each headphone performs on different humans. So a better solution might be having multiple heads with different shapes and sizes, but that requires a lot of work and money!

 

[outerspace]

The bigger issue in my opinion is that none of the current dummy heads are good enough. They have issues in both the bass and treble ranges. We tried to accomodate for the bass issues by measuring them on multiple humans, but the treble range issue is much harder to solve.

Is this related to IE headphones only or to AE too?

 

[SamV]

Is this related to IE headphones only or to AE too?

I would say all types of headphones pose problems for the dummy heads, but each type poses a different problem. Not to forget, there are also bone conduction headphones, which are getting more popular. Those are the hardest to properly measure at the moment (actually impossible to properly measure).

 

[solderdude]

The bigger issue in my opinion is that none of the current dummy heads are good enough. They have issues in both the bass and treble ranges. We tried to accomodate for the bass issues by measuring them on multiple humans, but the treble range issue is much harder to solve. For example, certain headphones show a noticeable dip in the treble range when compared to the target, but when you listen to it you can't really hear it. And when the dip fixed is with EQ, the headphone sounds too bright and much worse. Yet, there are other headphones that show a similar dip in the treble range, which you can hear and fix with EQ. I don't think the 5128 has solved that issue either. It may just be related to how each headphone performs on different humans. So a better solution might be having multiple heads with different shapes and sizes, but that requires a lot of work and money!

Hi Sam, good to see you here. Welcome at ASR.

 

[edahl]

Sure, studies have shown that the latest Harman target is preferred by the majority, but that doesn't mean everyone likes that target. We had discussions at RTINGS 2-3 years ago about supporting multiple targets or even allowing users to create their own target, but that would just add too much complexity to the website. I'm not working there anymore, so not sure what their current plans are. But the company is going through a period of expansion now, so any major changes to the review structure will probably happen only after that. I also believe they still prefer simplicity over complexity, so that feature may never appear.

The bigger issue in my opinion is that none of the current dummy heads are good enough. They have issues in both the bass and treble ranges. We tried to accomodate for the bass issues by measuring them on multiple humans, but the treble range issue is much harder to solve. For example, certain headphones show a noticeable dip in the treble range when compared to the target, but when you listen to it you can't really hear it. And when the dip fixed is with EQ, the headphone sounds too bright and much worse. Yet, there are other headphones that show a similar dip in the treble range, which you can hear and fix with EQ. I don't think the 5128 has solved that issue either. It may just be related to how each headphone performs on different humans. So a better solution might be having multiple heads with different shapes and sizes, but that requires a lot of work and money!

What happens if one measures the same headphone across different fixtures? I suppose it's a lot more complicated than headphones varying by a set function of frequency, say f, going from one head another. I.e., FR with respect to head A = f * FR with respect to head B.

 

[solderdude]

What happens if one measures the same headphone across different fixtures?

They will all differ. Such tests already have been done. Also Amir measured the exact same headphones on 5128 and 45CA.

 

[SamV]

What happens if one measures the same headphone across different fixtures? I suppose it's a lot more complicated than headphones varying by a set function of frequency, say f, going from one head another. I.e., FR with respect to head A = f * FR with respect to head B.

Yeah, the difference is not linear. We tried to come up with a transfer function few years ago by sending some of our headphones to Harman to measure on their 45CA. The differences between the measurements of different headphones were similar, but not identical. So couldn't come up with a function. One thing I've learned in evaluating headphones is that there's rarely a certainty about a measurement result, only averages. Speakers are much easier that way.

 

[edahl]

Yeah, the difference is not linear. We tried to come up with a transfer function few years ago by sending some of our headphones to Harman to measure on their 45CA. The differences between the measurements of different headphones were similar, but not identical. So couldn't come up with a function. One thing I've learned in evaluating headphones is that there's rarely a certainty about a measurement result, only averages. Speakers are much easier that way.

That's very fascinating. Thanks for the insight.

 

[outerspace]

there are also bone conduction headphones, which are getting more popular. Those are the hardest to properly measure at the moment (actually impossible to properly measure).

You can measure subjective deviation between in-ears eq'ed to harman target and bone conduction headphones by balancing right and left channel at different frequencies, if one channel play signal from bone conduction "driver" and another from this reference in-ear headphones. I think you can do it with AE headphones too with some modifications (e.g. removing one cup). Result would not be precise but up to +/- 2 dB (depend on the channel imbalance sensitivity at different frequencies).

 

[Deleted member 16543]

Yeah, the difference is not linear. We tried to come up with a transfer function few years ago by sending some of our headphones to Harman to measure on their 45CA. The differences between the measurements of different headphones were similar, but not identical. So couldn't come up with a function. One thing I've learned in evaluating headphones is that there's rarely a certainty about a measurement result, only averages. Speakers are much easier that way.

That's the famous geometric boundaries I've been harping on about here and there. Different geometry, different measurement... AND different appropriate target curve. However, at lower frequencies (good seal aside) those differences should become more and more equal for every set of headphones.
A few posts ago (#210) solderdude mentioned an experiment he did which showed that not being the case. Actually, he tested same headphones, same rig geometry but just different capsule and got results that didn't differ by the same amount between 3 sets of headphones.
I'm still trying to figure out what physics principles might be at play there to justify those results, other than different amount of seal and non airtight capsule placement, both of which he swears were not the case. Could you chime in on that?

EDIT: It just occurred to me that the seal is not only the one between pad and cheek. @solderdude, was one of those headphones open back, by any chance?