Master Complaint Thread About Headphone Measurements

[bobbooo]

A few posts ago (#210) solderdude mentioned an experiment he did which showed that not being the case. Actually, he tested same headphones, same rig geometry but just different capsule and got results that didn't differ by the same amount between 3 sets of headphones.
I'm still trying to figure out what physics principles might be at play there to justify those results, other than different amount of seal and non airtight capsule placement, both of which he swears were not the case. Could you chime in on that?

The physics principle at play here is acoustic impedance. Non industry-standard rigs are not accurate (and the error isn't even constant and compensatable) due to the coupler not having the same acoustic impedance as the human ear, resulting in such couplers interacting in an inconsistent way with the acoustic impedance of different headphones (and even pads). This is all explained (and demonstrated with measurements) by Oratory in this excellent Reddit post.

 

[Jimbob54]

Not a complaint but a suggestion . Can any wireless and/ or ANC phones be tested passive wired, wireless ANC off, AND wireless ANC on. The last one especially compared to passive wired.

 

[SamV]

You can measure subjective deviation between in-ears eq'ed to harman target and bone conduction headphones by balancing right and left channel at different frequencies, if one channel play signal from bone conduction "driver" and another from this reference in-ear headphones. I think you can do it with AE headphones too with some modifications (e.g. removing one cup). Result would not be precise but up to +/- 2 dB (depend on the channel imbalance sensitivity at different frequencies).

I guess, but the whole point is trying to find an objective and reliable way.

 

[SamV]

That's the famous geometric boundaries I've been harping on about here and there. Different geometry, different measurement... AND different appropriate target curve. However, at lower frequencies (good seal aside) those differences should become more and more equal for every set of headphones.
A few posts ago (#210) solderdude mentioned an experiment he did which showed that not being the case. Actually, he tested same headphones, same rig geometry but just different capsule and got results that didn't differ by the same amount between 3 sets of headphones.
I'm still trying to figure out what physics principles might be at play there to justify those results, other than different amount of seal and non airtight capsule placement, both of which he swears were not the case. Could you chime in on that?

EDIT: It just occurred to me that the seal is not only the one between pad and cheek. @solderdude, was one of those headphones open back, by any chance?

I'm not 100% sure myself but acoustic impedance, as mentioned, is probably one of the factors. Now add to this the research that showed humans compensate for their own HRTFs (and can even adapt to a new HRTF over time). So although we have different head/ear shapes and sizes, we more or less perceive the same sound. Does this simplify or complicate things? Should we just measure headphones in free-field because the head/ear shape and size effect is comepnsated for by the human? I haven't done enough experiments to know the answer.

 

[Deleted member 16543]

Acoustic impedance is another way of calling the cumulative effect of the internal reflections and absorptions, with their constructive or destructive interferences, and the effect of the air mass (distributed parameters), and boiling them all down into a lumped parameter. But this simplification (the lumping of all these physical phenomena into one single value) is valid only for low frequencies. Which is why the previous version of the geometrically inaccurate (but close enough as far as acoustic impedance) of the GRAS 43AG was OK to measure up to a certain frequency... but not higher than that.
Nevertheless, for low frequencies, the lumped parameter is a good approximation.. which is why I'm surprised to hear that different headphones don't measure all 'about' the same on two different rigs, the only difference being a 'lumped' EQ adjustment dependent on the rigs mutual differences, but not on the headphones.
In other words, at low frequencies it should be possible to find a correcting factor that predicts the low frequency response of a pair of headphone on rig A having measured it only on rig B, and do the same for other pairs of headphones using exactly the same correcting factor.

Unless the volume of the chamber comprised of rig+headphones, which varies with different headphones, operates in a non linear way to the lumped acoustic impedance value determination.
This is where a mechanical engineer with good knowledge of fluid dynamics would come in handy, I guess.

 

[Judas]

I totally understand about the target curve.
That's the difficult part, but also the most rewarding, I think.
This all ties up in a big circle to me and the work I'm doing.
I started with building a binaural microphone, but then I needed to equalize it.
Equalize to what? I needed to find out what a binaural microphone measures at the listening position of a well balanced pair of speakers.
To do that I needed very balanced speakers in quasi-anechoic listening conditions (still working on that although I have been using a working prototype for years now).
For music recordings I made do with off the shelf speakers and off the shelf EQ plugins, but that obviously won't do for precision measurements.

In all of this my main goal shifted from making the most accurate binaural microphone possible to making the most accurate pair of speakers possible (I am much more of a music listener than of a recordings maker, after all).
When the speakers will finally be ready it won't be much of a problem for me to measure them with my binaural microphone and share the target.
It will be a personalized (to the specific binaural microphone) target, but I think still educational to compare with other target curves of other rigs.

have you heard of Hooke audio? It is a Kickstarter company that is all in on Binaural (360 degree) recording. They have binaural microphone that plugs into your mobile device and is warn on your head.

 

[Deleted member 16543]

have you heard of Hooke audio? It is a Kickstarter company that is all in on Binaural (360 degree) recording. They have binaural microphone that plugs into your mobile device and is warn on your head.

Never heard of them. Now that I have, that seems like a core sound "binaural" set, only made easier to wear.
I think there's a few other companies that have played with the same concept.
This one looks really well made and convenient to use, at first glance.
These types of wearable "binaural" sets are not my cup of tea though.

 

[Szakuviron]

https://reference-audio-analyzer.pro/en/

Ihmo the best site with headphones measurements.

 

[amirm]

Not a complaint but a suggestion . Can any wireless and/ or ANC phones be tested passive wired, wireless ANC off, AND wireless ANC on. The last one especially compared to passive wired.

You mean repeat all measurements this many times? If so, no, I need a big raise before I sign up for that much work.

 

[amirm]

In other words, at low frequencies it should be possible to find a correcting factor that predicts the low frequency response of a pair of headphone on rig A having measured it only on rig B, and do the same for other pairs of headphones using exactly the same correcting factor.

I think the protocol is the problem with this. To the extent you can never really know if you have mounted a headphone correctly on the rig, any correction you create will have that much error in it when applied to other headphones. I think you mentioned the same with respect to bass leakage.
My sense is that this work is complicated enough to not have to then deal with calibration to another rig/standard! Even if you got it right, you will spend a lifetime explaining that it is correct!

 

[Deleted member 16543]

I think the protocol is the problem with this. To the extent you can never really know if you have mounted a headphone correctly on the rig, any correction you create will have that much error in it when applied to other headphones. I think you mentioned the same with respect to bass leakage.
My sense is that this work is complicated enough to not have to then deal with calibration to another rig/standard! Even if you got it right, you will spend a lifetime explaining that it is correct!

But assuming an always ideal seal (just for the sake of argument, of course), is the positioning of the headphones that critical, at low frequencies?
How does the volume of air affect the bass response, and how does the shape of that volume change that?
I'm well aware that at frequencies with wavelength comparable with the dimensions of the geometry the shape is very important, otherwise I wouldn't have wasted all that time making an anatomically accurate replica of the ear canal.. Still, at low frequencies I think things should be more simplified, and allow for that lumped correction factor to be used and applied (under the assumption of an ideal seal).

 

[Jimbob54]

You mean repeat all measurements this many times? If so, no, I need a big raise before I sign up for that much work.

Fair point. I would have said for the wireless ANC cans the comparisons most likely buyers would want to make is to other such cans. So wired passive measurements probably don't tell those users much about real world.

What I don't know is how much users of these favor ANC on or off (or indeed if engaging it significantly changes its response). But I'd definitely want to compare the Bose to the equivalent Sonys etc in wireless operation, not wired. I don't know how possible getting your test signals into them in this way is, let alone how much more onerous the testing would be, so apologies. Feel free to grant yourself a payrise though.

 

[solderdude]

https://reference-audio-analyzer.pro/en/

Ihmo the best site with headphones measurements.

No, certainly not. There is value to the website though. Just like there is value to other websites.

 

[SamV]

I think the protocol is the problem with this. To the extent you can never really know if you have mounted a headphone correctly on the rig, any correction you create will have that much error in it when applied to other headphones. I think you mentioned the same with respect to bass leakage.
My sense is that this work is complicated enough to not have to then deal with calibration to another rig/standard! Even if you got it right, you will spend a lifetime explaining that it is correct!

This problem can be solved for the most part on the headphone's side. A bunch of ANC headphones use the internal microphone to check for the seal and compensate for it. QC35 II and XM4 already do it. I just hope more headphones do it in the future (you don't even necessarily ANC for it). That's one of the things that bothered me about the HD 820. For a headphone that expensive I would expect a consistent bass, but it was one of the worst headphones we ever measured in that regard. Maybe instead of a glass back put a couple of microphones and some active electronics inside to fix that issue?

 

[Szakuviron]

No, certainly not. There is value to the website though. Just like there is value to other websites.

RAA have a lot measurements and measurements in self are pretty good (plus you have, a lot of types).

 

[solderdude]

They aren't pretty good at all. The compensation they use is wrong and way too smoothed.
The " Perception of headphone frequency response curves of equal loudness ISO 226-2009 ' plot is utter nonsense.
The output resistance plots, impedance plots on the other hand are really good.
A mixed bag. Some good and some bad stuff.

Some of their lists, calculators, and device measurements are very good though.

 

[Bjpenn88]

Amir, my suggestion would be that the measurements and recommendations much more useful if you compared and commented on headphones in the context of price. For example a cheap $30 headphone that is better than most $200 headphones would be useful to know. Similarly if a headphone is $200 and is only good in comparison to other $200 headphones, then a headphone that is $200 and is comparable to $1000 headphones would be in a different league. Maybe a better ranking / rating system could be used?
Also it would be good to make some comparisons of each headphone against the "reigning champ" of the price range - so we could know what the best headphones are currently available within each price category. I would also recommend the same for DACs/Amps as well

 

[zowper]

First off, I'm super grateful for the information and measurements you've supplied! You are a rock star! I would like to suggest three things (similar to many other suggestions as I've skimmed this thread):

1- Please include the "release price" in parenthesis next to the product names in the bar graphs. I find myself searching different products that scored similarly to the ones I'm looking at only to find that many of them are 10 times more expensive. Including the price in those comparison charts would allow me to search for the ones most interesting to me (either up or down in price).

2- Is there some kind of "sticky article" that has the current ranking of products for each test? A simple leaderboard (top 10 products for each test) would be amazingly helpful to guide my search!

3- Out of all of the reviewers and testers that I've used in my search, I would like to click your affiliate link the most before purchasing. Could you please create an affiliate link to the Amazon.com home page to include next to your donation link? I would even use it for non-audiophile purchases! (see: https://affiliate-program.amazon.com/help/node/topic/GWJ4AHCH7U5LCL86 and https://affiliate-program.amazon.com/help/node/topic/G9SMD8TQHFJ7728F ).

Thanks again for everything! You're the best!

 

[Robbo99999]

Amir, my suggestion would be that the measurements and recommendations much more useful if you compared and commented on headphones in the context of price. For example a cheap $30 headphone that is better than most $200 headphones would be useful to know. Similarly if a headphone is $200 and is only good in comparison to other $200 headphones, then a headphone that is $200 and is comparable to $1000 headphones would be in a different league. Maybe a better ranking / rating system could be used?
Also it would be good to make some comparisons of each headphone against the "reigning champ" of the price range - so we could know what the best headphones are currently available within each price category. I would also recommend the same for DACs/Amps as well

One thing to keep you going before/if any changes are made, you can use this list of headphones and rank them according to price and also whether they are recommended (Yes/No): https://www.audiosciencereview.com/forum/index.php?pages/HeadphoneReview/

One of the current acknowledgements by people-in-the-know (as shown by research), is that price is not indicative of preference......so price does not have a correlation with sound quality, which in some respects negates the relevance of having price-tiered reviews/awards. I think the best approach is scouring the index I linked for recommended headphones and then choosing one that is likely to physically fit you with no issues, and also take a look at the frequency response graph & distortion graphs and use those altogether to decide what you're gonna buy. I chose all my headphones (apart from my first headphone, K702) by using a similar approach (albeit using measurements made by Oratory1990 https://www.reddit.com/r/oratory1990/wiki/index/list_of_presets, as well as distortion measurements and physical dimension information from solderdude's website https://diyaudioheaven.wordpress.com/headphones/measurements/nad-viso-hp50/).

 

[rxp]

This problem can be solved for the most part on the headphone's side. A bunch of ANC headphones use the internal microphone to check for the seal and compensate for it. QC35 II and XM4 already do it. I just hope more headphones do it in the future (you don't even necessarily ANC for it). That's one of the things that bothered me about the HD 820. For a headphone that expensive I would expect a consistent bass, but it was one of the worst headphones we ever measured in that regard. Maybe instead of a glass back put a couple of microphones and some active electronics inside to fix that issue?

Sony's been doing that since the 1000x series. Apple actually introduced measuring ear canal length to the Air Pods Pro ,so even with IEM's where seal is typically easy the ear canal length is the other factor. That's why I find rtings frequency response consistency measure extremely useful. They use binural mics on real human subjects.