Suggestion: Measurement rig standardization between reviewers

[whazzup]

Given that measurement rigs are much more prevalent, it should make sense that specific units are shipped around so that we can see just how much deviation exists between rigs, instead of claiming one reviewer's measurements are better than another's.

For example, some iems sharing between amir and crinacle. They're tiny and cheap to ship across the world.

 

[ADU]

Howdy whazzup.

I think if you want folks to get on board with this, you'll need to be a little more specific about what your goals would be for this standard.

There are some standards already in place for the designs of different kinds of ear simulators or couplers. And there is of course the Harman curve, which can be useful up to a point for assessing (and performing some adjustments to) a headphone's tonal balance.

From my (and some other noteworthy folks') perspective, it is more difficult (but not necessarily impossible) to achieve reliable standards for IEMs than for over-ear headphones. Because IEMs bypass all of a person's head and pinna related effects. And need to be tailored more to a person's individual HRTF and in-ear impedance. That seemed to be one of the main points of this recent Canjam presentation by the noted Sennheiser engineer, Axel Grell (designer of the HD800, HD600, etc.)...

 

[ADU]

Jude also made a few similar points about the difficulties in doing reliable IEM measurements particularly with the older 711 couplers in his presentation from a couple years ago on the new HBK 5128 measurement system...

 

[ADU]

I don't use IEMs btw. So it would be interesting to hear some more opinions on the above from other users and experts on them, like Earfonia and Crinacle, for example.

 

[whazzup]

'Standardization' may not be the correct word. But right now reviewers tend to operate as islands. Each has a database of measurements, but users have no way to cross reference them.

But such a situation can be easily improved. By just sharing a few 'reference' units between reviewers and comparing the measurements of the exact same iem, we can easily suss out the deviations in measurements between each reviewer's rig. And I believe there're enough capable mathematicians / engineers to take such measured deviations and derive a generalized formula to allow, for example, Crinacle's iem measurements to be directly compared with Amir's measurements.

 

[617]

Are IEMs more variable or less variable when mounting them to an ear? I know that seating differences on a mannequin head can swing bass quite a bit, but are IEMs more repeatable?

 

[whazzup]

Are IEMs more variable or less variable when mounting them to an ear? I know that seating differences on a mannequin head can swing bass quite a bit, but are IEMs more repeatable?

Good question, for which I have no answer, hopefully someone more capable can drop by.
Still, I'm interested in how much variation there can be when the exact same iem is measured on different rigs.

 

[SIY]

There's a pile of variation when one set of headphones is measured on the same rig, but removed and replaced. But for some reason, no-one shows error bars based on a real gauge R&R study. I haven't measured IEMs yet, but they could very well be just as inconsistent.

 

[amirm]

There's a pile of variation when one set of headphones is measured on the same rig, but removed and replaced. But for some reason, no-one shows error bars based on a real gauge R&R study. I haven't measured IEMs yet, but they could very well be just as inconsistent.

That is why I show two channels to also cover unit variations.

 

[threni]

Given that measurement rigs are much more prevalent, it should make sense that specific units are shipped around so that we can see just how much deviation exists between rigs, instead of claiming one reviewer's measurements are better than another's.

For example, some iems sharing between amir and crinacle. They're tiny and cheap to ship across the world.

Likewise it would be instructive to have different units tested; I remember accusations (baseless, as it turned out) of 'golden samples' when Amir started testing Topping DACs. I'm more interested in that than a benchmark of testers' competence (especially as ASR continues to get more popular and manufacturers send him kit directly.) If we start to see inexplicably different results from different units then it makes sense to have multiple people test that unit.

 

[MayaTlab]

Are IEMs more variable or less variable when mounting them to an ear? I know that seating differences on a mannequin head can swing bass quite a bit, but are IEMs more repeatable?

One source could be Rtings. For IEMs their consistency graph shows five individual seatings on their own HATS. Just note that it's quite different from G.R.A.S. or B&K fixtures.
Ex :

Source :

Headphones v1.5 - Graph Tool

www.rtings.com

I don't know if that is representative of variation on real humans or not.

Measurements on real humans are difficult enough for over-ears, for in-ears they're massively difficult... unless you have access to the feedback mics some IEMs / earbuds use :

Source :

AirPods Pro Teardown

Apple further expands their collection of professional equipment, this time with a set of AirPods Pro. Apple’s line of wireless buds certainly has...

www.ifixit.com

Just note that what these mics measure isn't equivalent to measurements at the eardrum at higher frequencies.

Harman's articles on IEMs used a modded Sennheiser Momentum with a MEMs inside the output canal, to make sure that proper seal was attained :

They showed one graph of what several seatings looked like on one subject :

(Important to note that such measurements are only valid up to around, what, 800-1k Hz ? I'm not certain)
Source : https://www.aes.org/e-lib/online/browse.cfm?elib=18369

It's one of the small handful of real humans measurements I've seen for IEMs, they're very rare (publicly available that is). And even more so at the eardrum.

 

[MayaTlab]

Still, I'm interested in how much variation there can be when the exact same iem is measured on different rigs.

I won't @ him, but you may want to PM SamV (ex-Rtings) : https://www.audiosciencereview.com/forum/index.php?members/samv.27519/
He's compared a few IEMs on both a 5128 and Rtings's HATS and may be willing to send you the files.