Should we (I) get into speaker testing & measurement

[Wombat]

So my question is do we have enough people on this site to fund this? How much do I need to donate? It sounds like Amir needs $70K.
Speakers=yes

Forum funding for relatively inexpensive dacs is poor. Better for HPs and LSs? Hard to see. Just being realistic.

Exploiting the forums .com status could be the way but members may resist.

 

[SIY]

@Mad_Economist The systems I used were the Audio Precision AECM206 with two different supplied pinnae (each of which gave significantly different results), the same systems without the pinnae (even more different, but arguably not a fair test), and the miniDSP EARS.

 

[Daverz]

I think you need to start testing phono cartridges. Nobody tests phono cartridges anymore.

 

[andreasmaaan]

Some commentary on headphone measurement, since this is an area of immense personal significance to me:

Headphones are (properly) measured on anthropomorphic fixtures - either Head And Torso Simulators (HATS) per IEC60318-7 and ITU-T P58, or cheaper "ear and cheek" or "hearing protection test fixtures" which feature an anthropomorphic ear on a flat mounting plate such as GRAS' 43AG or 45CA. In either case, an anthropomorphic human pinna based on a population average (examples and requirements are given in ITU-T P57, IEC60318-7, and IEC60268-7, although to my knowledge only GRAS produces a pinna from the last) with a short "canal extension" tube leading from its ear canal entrance is mated with an IEC60318-4/IEC711 "ear simulator", which emulates the impedance of the human ear at the drum. These systems are used by most professionals measuring headphones.

Some have suggested in this thread that it is necessary to know the HRTF of a specific user to characterize headphone response. There may certainly be some truth here - the results of Smyth's excellent Realizer certainly could indicate in this direction - but I feel that it appears to be taken too far as currently interpreted. When the HRTFs of HATS systems have been measured, they fell relatively close to population averages (as they were intended to), and measurements done using HATS systems and analogous ear simulators have produced robust predictions of subjective frequency response and preference - as an example, the work of Sean Olive, which was done primarily on a GRAS 43AG and 45CA, fairly reliably predicts user preference for headphones based on frequency response.View attachment 28415
Consequently I feel it's quite hard to argue that the measurement of headphones is a truly uncertain space. There are certainly nuances - variations in both placement and individual anatomy may influence results to some degree - and I would say that it is a less filled in space than the world of speakers, but it's an area where we have fairly strong tools and a reasonable understanding of how to use them. It's a developing area, without question, and one that was long neglected in favour of speakers, but we have a reasonable body of literature to draw upon at the moment (Olive's headphone paper collection bundles many of the major ones), and can come to fairly robust conclusions.

@SIY you mention that changing systems significantly changed the response of the headphones you were measuring - may I ask what systems you were comparing? While in principle I would expect some variation based on the parameters of the specific pinnae used - something on the order of the differences between the major brands' HATS perhaps, which I would expect to be largely (although perhaps not completely) accounted for by using a DF-HRTF appropriate to the system in question as compensation - but I wouldn't expect it to be very large. Although if you're mostly measuring speakers and I'm mostly measuring headphones, perhaps we have different definitions of large frequency response variation

Thanks for the detailed post. This possibly explains why, despite the wide variations in (raw) measured response from site to site, I've always found Innerfidelity's measurements - compensated by the Harman curve, rather than Innerfidelity's own (old) compensation curve - to most closely correspond to my own preferences.

Or to put it another way: The Harman curve is IME valid, but only when implemented with reference to the raw response obtained from a GRAS 43AG rig.

 

[Mad_Economist]

@Mad_Economist The systems I used were the Audio Precision AECM206 with two different supplied pinnae (each of which gave significantly different results), the same systems without the pinnae (even more different, but arguably not a fair test), and the miniDSP EARS.

Ahh, that makes sense. The EARS is completely noncompliant with all standards and kind of flawed - it could probably be used for some purposes, with some work, but its raw output will not mirror that of a proper system (both due to its inaccurate pinnae and lack of ear impedance emulation), and its supplied compensation data is subpar.

For the AECM206 ears, I take it that you were testing a headphone which had shallow enough pads to sit against the pinnae? To my understanding, hardness is the only meaningful difference between its two pinnae, following to some degree from Welti's paper on headphone coupling and HATS/headphone test fixture pinna stiffness. The current version of the IEC60318-4 standard accepts hardness in the range of Shore OO-30-60 - I believe most manufacturers are standardizing around 35 for the less stiff pinnae, which are preferred for headphone measurements.

As you say, without the pinnae, it's not a fair test - the pinna's presence in the earpad is core to the response of the headphone at the eardrum for obvious reasons.

Thanks for the detailed post. This possibly explains why, despite the wide variations in (raw) measured response from site to site, I've always found Innerfidelity's measurements - compensated by the Harman curve, rather than Innerfidelity's own (old) compensation curve - to most closely correspond to my own preferences.

Or to put it another way: The Harman curve is IME valid, but only when implemented with reference to the raw response obtained from a GRAS 43AG rig.

Bear in mind that the Harman curve - and all compensation targets - incorporate an HRTF element which is specific to the head(s) used. In the case of Harman, that's a styrofoam head around a GRAS 45CA, and I would not recommend directly applying the Harman target to any measurements from systems not using GRAS's pinna - although the pinna is nominally standardized, the method of standardization allowed substantial variation between the three main manufacturers (Head Acoustics, GRAS/previously Knowles, and Brüel & Kjær), with according variation in HRTF.

The Harman target in particular is fairly smoothed due to the circumstances of its derivation, and so may be more suitable for this purpose than some more granular HRTFs, but I would be cautious in particular with applying it to Head Acoustics HATS data, as their HATS has been shown to fail to meet the IEC HRTF requirements.

As an aside, Innerfidelity's old Independent-Of-Direction curve is quite an oddity, and I'm not entirely sure of why Tyll chose it - his previous measurements at Headroom used the industry-standard diffuse field HRTF target, but for some reason when moving to Innerfidelity he pivoted to Head-Acoustics' eccentric and nonstandard compensation, which changes the look of the centre of the midrange.

 

[Blumlein 88]

Some commentary on headphone measurement, since this is an area of immense personal significance to me:

Headphones are (properly) measured on anthropomorphic fixtures - either Head And Torso Simulators (HATS) per IEC60318-7 and ITU-T P58, or cheaper "ear and cheek" or "hearing protection test fixtures" which feature an anthropomorphic ear on a flat mounting plate such as GRAS' 43AG or 45CA. In either case, an anthropomorphic human pinna based on a population average (examples and requirements are given in ITU-T P57, IEC60318-7, and IEC60268-7, although to my knowledge only GRAS produces a pinna from the last) with a short "canal extension" tube leading from its ear canal entrance is mated with an IEC60318-4/IEC711 "ear simulator", which emulates the impedance of the human ear at the drum. These systems are used by most professionals measuring headphones.

Some have suggested in this thread that it is necessary to know the HRTF of a specific user to characterize headphone response. There may certainly be some truth here - the results of Smyth's excellent Realizer certainly could indicate in this direction - but I feel that it appears to be taken too far as currently interpreted. When the HRTFs of HATS systems have been measured, they fell relatively close to population averages (as they were intended to), and measurements done using HATS systems and analogous ear simulators have produced robust predictions of subjective frequency response and preference - as an example, the work of Sean Olive, which was done primarily on a GRAS 43AG and 45CA, fairly reliably predicts user preference for headphones based on frequency response.View attachment 28415
Consequently I feel it's quite hard to argue that the measurement of headphones is a truly uncertain space. There are certainly nuances - variations in both placement and individual anatomy may influence results to some degree - and I would say that it is a less filled in space than the world of speakers, but it's an area where we have fairly strong tools and a reasonable understanding of how to use them. It's a developing area, without question, and one that was long neglected in favour of speakers, but we have a reasonable body of literature to draw upon at the moment (Olive's headphone paper collection bundles many of the major ones), and can come to fairly robust conclusions.

@SIY you mention that changing systems significantly changed the response of the headphones you were measuring - may I ask what systems you were comparing? While in principle I would expect some variation based on the parameters of the specific pinnae used - something on the order of the differences between the major brands' HATS perhaps, which I would expect to be largely (although perhaps not completely) accounted for by using a DF-HRTF appropriate to the system in question as compensation - but I wouldn't expect it to be very large. Although if you're mostly measuring speakers and I'm mostly measuring headphones, perhaps we have different definitions of large frequency response variation

Weren't those tests of rating 32 headphones actually emulating headphone measurements in a single headphone? Seems like a big whoops is that really the best way? You need to close the circle of confusion on your test phone first. It has to be the most transparent and best performing if you emulate other phones. It may indeed indicate the other factors are less relevant than just the basic response presented to the ear, but I've not seen work to convince me of that so much yet.

 

[andreasmaaan]

Bear in mind that the Harman curve - and all compensation targets - incorporate an HRTF element which is specific to the head(s) used. In the case of Harman, that's a styrofoam head around a GRAS 45CA, and I would not recommend directly applying the Harman target to any measurements from systems not using GRAS's pinna - although the pinna is nominally standardized, the method of standardization allowed substantial variation between the three main manufacturers (Head Acoustics, GRAS/previously Knowles, and Brüel & Kjær), with according variation in HRTF.

I'm not sure I follow you here. Yes, all compensation curves must incorporate a HRTF specific to the rig used - hence my comment about the (IME) narrow applicability of the Harman target to the specific GRAS rig.

Are you correcting this statement, adding additional information, or simply agreeing?

 

[Mad_Economist]

Weren't those tests of rating 32 headphones actually emulating headphone measurements in a single headphone? Seems like a big whoops is that really the best way? You need to close the circle of confusion on your test phone first. It has to be the most transparent and best performing if you emulate other phones. It may indeed indicate the other factors are less relevant than just the basic response presented to the ear, but I've not seen work to convince me of that so much yet.

Prior to his work using the virtualization method using an equalized "simulator" headphone, Olive evaluated the procedure in a paper. Note that Olive is testing only the frequency response contribution to perceived sound quality - and only frequency response for a single case. Obviously, there are other factors to headphone sound (I would contend that "variation in perceive frequency response" is a figure of merit for headphone designers, as an example, and one that has a lot of potential issues to overcome), but the preference results and subjective assessments of the simulated headphones don't vary too much, and, key, variations in simulator headphones across Olive's papers (at minimum three different designs, the AKG K701, Sennheiser HD518, and Stax SR009, have been used, and possibly more) has not produced large variations in preferred response, so there's a limit to the effect of whatever we're excluding here.

I wouldn't argue that HATS measurements log perfectly to perceived frequency response in all cases, certainly - and I would argue that there is some skill involved in getting closer - but I think that the information we have from preference tests such as Olive's or Lorho's indicates that whatever variation exists there is not extremely large, and likely fairly small.

Edit: As far as why Olive is only evaluating frequency response, the data as stands in the field seems to strongly indicate that frequency response is the dominant factor in perceived headphone sound quality. Some papers, ex. Olive & Temme 2014 have investigated other factors, but these do not seem to be large compared to frequency response, which coheres with the data from speaker preference research IMO.

 

[Mad_Economist]

I'm not sure I follow you here. Yes, all compensation curves must incorporate a HRTF specific to the rig used - hence my comment about the (IME) narrow applicability of the Harman target to the specific GRAS rig.

Are you correcting this statement, adding additional information, or simply agreeing?

Largely agreeing and providing some additional context, although I was attempting to caution against applying the Harman data to the Head Acoustics data from Innerfidelity as well, as its HRTF is different from that of GRAS equipment, which I had read your comment as saying you were doing - my apologies if I've misread.

 

[andreasmaaan]

Largely agreeing and providing some additional context, although I was attempting to caution against applying the Harman data to the Head Acoustics data from Innerfidelity as well, as its HRTF is different from that of GRAS equipment, which I had read your comment as saying you were doing - my apologies if I've misread.

Ok thanks. And no in fact, you were providing useful additional info, which I see now

 

[andreasmaaan]

Largely agreeing and providing some additional context, although I was attempting to caution against applying the Harman data to the Head Acoustics data from Innerfidelity as well, as its HRTF is different from that of GRAS equipment, which I had read your comment as saying you were doing - my apologies if I've misread.

Further to these posts, how would you go about applying compensation based on a measured response from e.g. Head Acoustics? Are the relevant HRTFs published anywhere that you know of? In theory, one should be able to calculate a compensation curve based on knowledge of (1) the HRTF of Harman's rig, (2) the HRTF of a given measurement rig, and (3) the raw response of a given headphone on said rig, I would have thought?

 

[Mad_Economist]

Further to these posts, how would you go about applying compensation based on a measured response from e.g. Head Acoustics? Are the relevant HRTFs published anywhere that you know of? In theory, one should be able to calculate a compensation curve based on knowledge of (1) the HRTF of Harman's rig, (2) the HRTF of a given measurement rig, and (3) the raw response of a given headphone on said rig, I would have thought?

Bear in mind here that HRTF is specific to conditions - when we speak of HRTFs in the headphone case we're usually talking about HRTF in a diffuse field (DF-HRTF) because this was the standard headphone target based on Günther Theile's research in the 80s for decades, although prior it usually referred to the 0/0 (vertically and horizontally centered on the ear axis in front of the HATS) free field HRTF that was used as the target. Most manufacturers supply HRTF data with their HATS - ex. Tyll had frontal free field, diffuse field, and independent of direction data for his Head Acoustics HMS II.3, and I have diffuse field and free field from many axes and elevations data for my 4128C - and independent measurements have been done (such as Hammershøi & Møller 2008, which I linked above, as well as Snaidero, Jacobsen, & Buchholz 2011, and there have been a few others.

Since HATS are meant to be population averages, some also use population average HRTFs as targets (this is true of the Reddit user Oratory1990, who has a large database on a GRAS 43AG, for example) - a fairly comprehensive review of past aggregated HRTFs is presented somewhere in Hammershøi & Møller 2008.

The Harman target is distinct, technically, in that it wasn't derived in a sound field that has standardized characteristics like a diffuse or free field, however: it's the product of an average of several measurements of a pair of Revel F208s in Harman's reference listening room, equalized flat to a free field mic at the listening position (I believe it was a 5.1 or 7.1 system that was used for Olive & Welti 2012, actually, but all recent work uses a different "baseline"), which Olive & Welti then allowed users to adjust with a pair of shelf filters for their listening tests. In principle, the way to get a directly applicable result would probably be to, as Tyll Hertsens did, bring a Head Acoustics HATS to the Harman reference room and measure it with the same positioning as Olive & Welti used. However, the flat in-room response is rather analogous in shape to the DF-HRTF of a HATS (which, if you consider the high-frequency behavior of sound in rooms, makes some sense IMO), albeit quite a smoothed one, and I tend to be sympathetic to the position that you can get pretty close to Harman results by applying their preference test derived filters to the DF-HRTF of the HATS used for the headphone measurements - I should note that this is an original position without verification in the literature, however, although it is analogous to what RTings does to some degree.

 

[Blumlein 88]

Prior to his work using the virtualization method using an equalized "simulator" headphone, Olive evaluated the procedure in a paper. Note that Olive is testing only the frequency response contribution to perceived sound quality - and only frequency response for a single case. Obviously, there are other factors to headphone sound (I would contend that "variation in perceive frequency response" is a figure of merit for headphone designers, as an example, and one that has a lot of potential issues to overcome), but the preference results and subjective assessments of the simulated headphones don't vary too much, and, key, variations in simulator headphones across Olive's papers (at minimum three different designs, the AKG K701, Sennheiser HD518, and Stax SR009, have been used, and possibly more) has not produced large variations in preferred response, so there's a limit to the effect of whatever we're excluding here.

I wouldn't argue that HATS measurements log perfectly to perceived frequency response in all cases, certainly - and I would argue that there is some skill involved in getting closer - but I think that the information we have from preference tests such as Olive's or Lorho's indicates that whatever variation exists there is not extremely large, and likely fairly small.

Edit: As far as why Olive is only evaluating frequency response, the data as stands in the field seems to strongly indicate that frequency response is the dominant factor in perceived headphone sound quality. Some papers, ex. Olive & Temme 2014 have investigated other factors, but these do not seem to be large compared to frequency response, which coheres with the data from speaker preference research IMO.

Reading over it quickly they were developing this by having listeners do EQ to match the frequency balance heard over speakers. I don't have a disagreement that FR is a first order issue to get right. Only if you can do FR measures of headphones with current apparatus and have that really track with what people hear. Because the various test rigs give what appear to be rather different FR results.

 

[andreasmaaan]

In principle, the way to get a directly applicable result would probably be to, as Tyll Hertsens did, bring a Head Acoustics HATS to the Harman reference room and measure it with the same positioning as Olive & Welti used.

In other words, then, and of course in the absence of knowledge of a specific listener’s HRTF, the Harman curve is the correct curve to apply to measurements taken on Innerfidelity’s HATS?

 

[Mad_Economist]

Reading over it quickly they were developing this by having listeners do EQ to match the frequency balance heard over speakers. I don't have a disagreement that FR is a first order issue to get right. Only if you can do FR measures of headphones with current apparatus and have that really track with what people hear. Because the various test rigs give what appear to be rather different FR results.

Depending on your read of Theile 1986 (and possibly 2016? Whatever his recent paper was), the use of loudspeakers as a reference could vary from a moderate potential problem to a really big one due to the so-called "SLD effect" (it's somewhere toward the front of the Theile paper I linked above). Fortunately, unless I've missed something in Olive, that's not the case here - the "simulated" headphones and real versions were compared and the hierarchy of preference for listeners stayed largely the same, with Olive and Welti having credible explanations (ex. leak in one model) for the differences.

There may be some disparities between similarly-compensated standards-compliant measurement systems - this is a topic of significant personal interest to me, actually, as I attribute these potentially-extant differences some of the cause of disparities in perceived FR in headphones in the treble and upper midrange - but I haven't seen anything to date that indicates that they're large. Something that may influence your perception here is that there are both multiple compensation targets in current usage (diffuse field, Harman, as well as some "home made" targets for professional systems such as those used by RTings and GoldenEars), and a large number of measurements on non-compliant DIY or budget systems online.

 

[Mad_Economist]

In other words, then, and of course in the absence of knowledge of a specific listener’s HRTF, the Harman curve is the correct curve to apply to measurements taken on Innerfidelity’s HATS?

In short, no. The Harman curve is applicable to GRAS HATS, not the Head-Acoustics HATS used by Innerfidelity - if you desired to apply something Harmanish, my suggestion would be either applying the Harman shelf filters to the Head Acoustics' DF-HRTF (Tyll posted it in one of his old articles about HRTF, I recall), or follow Hertsens' strange Harman journey to its end and see if you can make something better from the raw data he posted than he did. I can offer some commentary on this whole strange affair at some later time if there's interest, although it's quite niche even within this discussion, and the TL;DR is that what he came away with isn't a valid approximation of Harman's compensation.

Edit: I should note that neither of the things I've suggested as options here necessarily has the predictive power in terms of personal preferences that Harman's work does - it might, and I'd put decent money on the preference-shelf-filter compensated DF-HRTF working well, but unlike a lot of the other stuff I'm saying here, I don't have a cite to point at.

 

[andreasmaaan]

In short, no. The Harman curve is applicable to GRAS HATS, not the Head-Acoustics HATS used by Innerfidelity - if you desired to apply something Harmanish, my suggestion would be either applying the Harman shelf filters to the Head Acoustics' DF-HRTF (Tyll posted it in one of his old articles about HRTF, I recall), or follow Hertsens' strange Harman journey to its end and see if you can make something better from the raw data he posted than he did. I can offer some commentary on this whole strange affair at some later time if there's interest, although it's quite niche even within this discussion, and the TL;DR is that what he came away with isn't a valid approximation of Harman's compensation.

Ok thanks for following up. I’m obviously looking for too simple an answer here as my interest in headphones is limited I appreciate your help though, and will come back to this with renewed time and interest at some point in the future...

 

[Mad_Economist]

Ok thanks for following up. I’m obviously looking for too simple an answer here as my interest in headphones is limited I appreciate your help though, and will come back to this with renewed time and interest at some point in the future...

Completely understandable, headphones are kind of a pain! If you have any questions, feel free to DM me, by the way - I usually lurk here since the conversation doesn't lean so much toward what I have anything to offer about most of the time, but I do check my inbox.

 

[SIY]

For the AECM206 ears, I take it that you were testing a headphone which had shallow enough pads to sit against the pinnae? To my understanding, hardness is the only meaningful difference between its two pinnae, following to some degree from Welti's paper on headphone coupling and HATS/headphone test fixture pinna stiffness. The current version of the IEC60318-4 standard accepts hardness in the range of Shore OO-30-60 - I believe most manufacturers are standardizing around 35 for the less stiff pinnae, which are preferred for headphone measurements.

I used several different types of headphones when I reviewed the AECM206. All gave different results with a change in the pinnae durometer, which I attribute mostly to the seal and flexure. Frankly, neither of them was right, and I believe that much of the issue there is trying to us a single durometer with isotropic bulk compressional characteristics, which does NOT well simulate an actual ear. Yes, they meet standards. And I think the standards are far off base.

Besides the eye-opening (so to speak!) experience with the Smyth, the final straw in my confidence about headphone measurement came when I tested the 1MORE over ear, which despite its name, is more of an on-ear headphone. No matter which pinnae I used, the bass response measured looked absolutely nothing like how the headphones sounded.

 

[Mad_Economist]

I used several different types of headphones when I reviewed the AECM206. All gave different results with a change in the pinnae durometer, which I attribute mostly to the seal and flexure. Frankly, neither of them was right, and I believe that much of the issue there is trying to us a single durometer with isotropic bulk compressional characteristics, which does NOT well simulate an actual ear. Yes, they meet standards. And I think the standards are far off base.

Besides the eye-opening (so to speak!) experience with the Smyth, the final straw in my confidence about headphone measurement came when I tested the 1MORE over ear, which despite it's name, is more of an on-ear headphone. No matter which pinnae I used, the bass response measured looked absolutely nothing like how the headphones sounded.

This has been a complaint in the lit for a while - I believe recall seeing it in a paper from the 90s, actually - although I believe that a reasonable read of Welti 2015 is that a sufficiently compliant pinna is reasonably close for most purposes. I'd prefer RTings-style multi-person in-ear low frequency measurements, as a rule, but this gets prohibitively time-consuming and costly pretty fast.

I will also note that placement of headphones is something that takes a while to get the hang of, in my experience - it took quite a while before I consistently achieved "typical" levels of seal on my HATS vs. on my head. Some people use noise or other continuous stimulus plus an RTA (or in Hertsens' case an oscilloscope and low frequency square wave) to help position for better coupling for this reason.

Edit: I'm probably at least partially biased by the fact that most of the headphones I measure are circumaural designs with fairly deep pads, vs. the more consumer average designs which tend to at least to some degree sit on the pinnae, as well, I'll admit. Equally, the pool of headphones that naturally couple very poorly to HATS doesn't overlap that much with the pool of headphones people typically seek out measurements of, I'd contend.

Edit 2 (I really need to slow down and think before I post): The ITU-R BS.708 standard for studio monitor headphones was based on Theile's above-linked research, and relies on ear-entrance measurements on real human ears, but unfortunately I'm not aware of anyone that's actually ever done testing under such circumstances. In general, real human ears have generally proven too expensive and annoying to work with, at least in groups, particularly while data of acceptable quality can be had from standard systems.