HBK Headphone Measurement Talks from Head-Fi and Sean Olive

[ADU]

I was just about to respond to @ADU by referencing the same studies and the same argument.

Well at least you're both on the same page.

I raised what I felt were some reasonable questions about the assumption that trained listeners are more discriminating than untrained listeners, based on the first study that GaryH posted on this. And rather than addressing any of those questions I was given a new set of graphs from another unidentified study that showed something different...

So be it... What I see in the graphs from this new study, with considerably more untrained listeners than trained listeners, is that the relatively small group of trained listeners are more consistent in their opinions and preferences than the much larger group of untrained listeners... Which is what I've said from the very beginning of this conversation.

I'm still waiting though for someone to actually define or articulate (without throwing another graph at me) what "more discriminating" actually means in this context... Dr. Olive seemed to be starting to do that in his post. But somehow his reply got cut off in mid-sentence...

Trained listeners have been shown to be more discriminating and consistent in rating loudspeakers and headphones. The How to Listen is useful in training them to identify

("identify" is where his comment seemed to end)

This is probably neither here nor there, but perhaps if the above sampling of trained listeners was also larger it would also show more inconsistencies as well, like the much bigger untrained group. (?)

 

[ifloatoveryou]

Let it be repeated here : the only quantifiable advantage so far observed (or at least published) with trained listeners is that they are more discriminatory and more consistent when evaluating speakers and headphones within the format of Harman's research. Very useful in that context, no doubts about it.

When evaluating speakers, untrained listeners and reviewers would give the same speaker different ratings, and trained listeners were much more consistent when evaluating the same speaker on different occasions. If you want I can find the paper and give more useful/detailed information than my opinion of "much more consistent", but I remember finding this part of the study compelling.

As an exercise it's an interesting thing to do, but probably shouldn't be seen as the be-all, end all thing GaryH thinks it is. You can cheat your way through some of tests BTW (the noise one for example).

I agree with this sentiment. I probably think being a trained listener is a more useful heuristic than you think it is, but at the end of the day we can agree that there are more important things to be concerned about when it comes to reviews.

 

[ifloatoveryou]

I am too, that's why I do it my way which differs from those of many others.
I found most reviews of headphones,that I know pretty well, to be all over the place and overly positive filled with advertising material. These are the usual suspects most of the case. Its the main reason you can only find me on my own website and here.
There is so many commercialism involved and maintaining good relationships with manufacturers (that because of this will send them samples) that I don't trust any of it unless they find the same things I do.

I am not sponsored, I rarely receive samples (because I do not hype what manufacturers want hyped) and I do have references to compare to.
My measurement method differs and have my reasons for that. Overlay them with those from Amir and they are very close.
Besides I also look for other measurements and reviews (when available) as well and am quite well trained in audio and listening.
That's why I do what I do and how I do this. People are free to read what I write and like or dismiss it for whatever reasons they have.



I get some positive feedback now and then but not by nerds.
I did get one hate mail once accusing me of snake-oil business (the passive filters which is the only thing I sell simply because I am the only one designing and making them).
Most nerds will not visit my site though as they know how I measure and don't agree.

I like your reviews are good and appreciate all the work you put into them. Trained listener or not is a relatively minor consideration that most people do not care about. I believe there is reason to go through with the program/train your ears but am not so bothered that you may elect not to.

I also believe there are many people who appreciate your work, even if you don't get very much positive feedback. I would imagine most people who are interested in and at least somewhat knowledgeable about the science behind headphones, and have seen your website, appreciate what you do.

 

[ADU]

I am curious why some folks here seem to be so invested in this concept of discrimination btw. And why that seems to be more important, or at least just as important as consistency. Perhaps someone could also try to explain that a little better.

 

[solderdude]

I am curious why some folks here seem to be so invested in this concept of discrimination btw. And why that seems to be more important, or at least just as important as consistency. Perhaps someone could also try to explain that a little better.

When you are able to discriminate deviations a bit better by knowing what to listen for, one may well be able to score headphone more consistently ?

This can be trained with the Harman program or on various other ways.
It's not that the Harman program will ensure reviews are better.
There can still be financial gain or the desire to receive more free samples or internet fame, relations with importers and manufacturers in play that want stuff hyped.
I believe @amirm also is honest in what he does and is not concerned about anything other than his measurements and evaluation.
There are also some other reviewers I think are honest b.t.w.

The 'desire' to see a specific test program being taken thus may be handy for research (to see if the participant can detect things) and or one's own piece of mind (to find out what you can and can't hear).
It will give absolutely NO guarantee a reviewer is trustworthy.

I would expect people just getting into the fascinating world of audio benefiting from such training where for experienced ones it would more be sort of a 'test/confirmation' for themselves.

 

[Robbo99999]

What Q on the shelf filters? Wondering if there's a "standard." I remember in one of the Harman papers they used "2nd order" filters with a frequency of 2.5khz as their treble tone control, but I don't know how to translate that to a shelf filter + q setting on my PEQ. Perhaps someone knows.

Ha, yes, sorry, I forgot to include the Q, it's Q0.5, with 3 High Shelf Filters at 63Hz / 632Hz / 6324Hz - near perfect linear tone control from 20-20000Hz, all gain values to be the same in each filter, only small changes required, eg 0.2dB changes at a time for each filter to notice a difference in tonality.

So terms and/or words cannot have more than one meaning? Toole should have made a completely new term named "the cycle of uncertainty" instead?

Do you realize how ridiculous you sound?

I think that's just how DualTriode thinks...

 

[Robbo99999]

I am too, that's why I do it my way which differs from those of many others.
I found most reviews of headphones,that I know pretty well, to be all over the place and overly positive filled with advertising material. These are the usual suspects most of the case. Its the main reason you can only find me on my own website and here.
There is so many commercialism involved and maintaining good relationships with manufacturers (that because of this will send them samples) that I don't trust any of it unless they find the same things I do.

I am not sponsored, I rarely receive samples (because I do not hype what manufacturers want hyped) and I do have references to compare to.
My measurement method differs and have my reasons for that. Overlay them with those from Amir and they are very close.
Besides I also look for other measurements and reviews (when available) as well and am quite well trained in audio and listening.
That's why I do what I do and how I do this. People are free to read what I write and like or dismiss it for whatever reasons they have.



I get some positive feedback now and then but not by nerds.
I did get one hate mail once accusing me of snake-oil business (the passive filters which is the only thing I sell simply because I am the only one designing and making them).
Most nerds will not visit my site though as they know how I measure and don't agree.

Well, I like your site, I'm aware of the limitations of the measurement rig you use, so I know what parts I can interpret from your frequency response measurements.....and your overall reviews of the headphones is useful incl channel matching / cup size / comments on build quality & fitment issues / seal effects on bass / distortion measurements.....so it's useful!

 

[Sean Olive]

Can you please comment on how this is done or what equipment is used?

Most audiometers don't measure beyond 8kHz. They are intended for measuring your ability to hear and understand speech. A high frequency audiometer allows measurements of your hearing thresholds up to 20kHz. They are also more expensive ($10k+),

 

[pozz]

high frequency audiometer

For example, those produced by Interacoustics?

That's such a straightforward answer.

I guess the real answer, for me, is to do my research. I have a persistent impression that HF audiometry just isn't done outside of experimental circumstances. Which obviously isn't true.

The relevant calibration standards I see are ISO 389-5 2004 or ANSI S3.6-2010.

I think Interacoustics relies on a specific headset. I would love to understand how it works or what's necessary in the procedure for consistent results.

 

[Sean Olive]

For example, those produced by Interacoustics?

That's such a straightforward answer.

I guess the real answer, for me, is to do my research. I have a persistent impression that HF audiometry just isn't done outside of experimental circumstances. Which obviously isn't true.

The relevant calibration standards I see are ISO 389-5 2004 or ANSI S3.6-2010.

I think Interacoustics relies on a specific headset. I would love to understand how it works or what's necessary in the procedure for consistent results.

I believe this is the model.. I didn't purchase it nor have I used it. The one we use in our lab is much older, cheaper and doesn't go beyond 8kHz

Clinical Audiometer | AudioStar Pro | Grason-Stadler

Test patients’ hearing with confidence using the GSI AudioStar Pro. Learn here about its features, which improve your clinical audiometry efficiency and data management workflow.

www.grason-stadler.com

 

[GaryH]

I raised what I felt were some reasonable questions about the assumption that trained listeners are more discriminating than untrained listeners, based on the first study that GaryH posted on this. And rather than addressing any of those questions I was given a new set of graphs from another unidentified study that showed something different...

That data on over-ear headphones is from this study, and it doesn't show anything different to the IEM data I posted previously, they are in accordance that trained listeners are more consistent and discriminating. I really don't know why there's so much confusion over the latter term. More discriminating means exactly that - they are more easily able to discriminate between sound quality (in these tests dependent on frequency response of headphones), which is reflected in the larger spread of ratings. I thought I explained this quite clearly in reference to the data in a previous post.

 

[edechamps]

Clinical Audiometer | AudioStar Pro | Grason-Stadler

Test patients’ hearing with confidence using the GSI AudioStar Pro. Learn here about its features, which improve your clinical audiometry efficiency and data management workflow.

www.grason-stadler.com

I've been wondering: if it's so difficult to accurately measure the actual response (at the eardrum) of headphones above 8 kHz, then how do we know we can trust this audiometry equipment? Wouldn't the manufacturer hit the exact same problem as soon as they try to calibrate their product? How come they can do accurate measurements up to 20 kHz but we can't?

 

[Robbo99999]

I've been wondering: if it's so difficult to accurately measure the actual response (at the eardrum) of headphones above 8 kHz, then how do we know we can trust this audiometry equipment? Wouldn't the manufacturer hit the exact same problem as soon as they try to calibrate their product? How come they can do accurate measurements up to 20 kHz but we can't?

I certainly don't know how they can create valid results up to 20kHz using headphones when small changes in seating variation creates massive changes to frequency response above 10kHz. But to offer an answer to your post (if I understand correctly how this functions), then the device doesn't measure at the eardrum, it just produces the sounds, and then the person wearing the headphones says whether they can hear it or not - the device records the hearing thresholds, based on the answers from the subject and the person who's conducting the testing & controlling the test tones (which are produced by the machine).

(Re calibration, I would think that would be problematical too, headphones are unnaturally positioned vs speakers so would affect how sound propagates down the ear canal (ultimately) & individual anatomy specific especially above 10kHz so unable to calibrate on a generic dummy head, so I would think testing using a speaker would be more valid (& real world naturally applicable) than using a headphone....and using a speaker would solve the seating variation issues of headphones that happen above 10kHz.)

 

[Sean Olive]

I've been wondering: if it's so difficult to accurately measure the actual response (at the eardrum) of headphones above 8 kHz, then how do we know we can trust this audiometry equipment? Wouldn't the manufacturer hit the exact same problem as soon as they try to calibrate their product? How come they can do accurate measurements up to 20 kHz but we can't?

Well the obvious answer is that getting a reproducible signal to the eardrum at those small wavelengths is challenging. Small changes in positioning of the headphones on the human ear and coupler etc have huge effects at those frequencies. So I'm sure the threshold tolerances are likely quite large for these reasons and the fact that thresholds are likely quite elevated at those HF frequencies. For in-ear headphones reproducible measurements at HF are challenging and will vary depending on the silicone tip used and whether it blocks the sound when inserted in an ear simulator or in some cases the inside section of the rubber pinna may block the HF.. One reason why we didn't use a pinna (just the coupler with a metal nozzle) when we did our IEC measurements for the research


Likely the HF headphones and the couplers they use for audiometry calibration are designed to minimize these variations in measurement. Removing the pinna from the measurement would be the first step.

 

[Sean Olive]

I certainly don't know how they can create valid results up to 20kHz using headphones when small changes in seating variation creates massive changes to frequency response above 10kHz. But to offer an answer to your post (if I understand correctly how this functions), then the device doesn't measure at the eardrum, it just produces the sounds, and then the person wearing the headphones says whether they can hear it or not - the device records the hearing thresholds, based on the answers from the subject and the person who's conducting the testing & controlling the test tones (which are produced by the machine).

(Re calibration, I would think that would be problematical too, headphones are unnaturally positioned vs speakers so would affect how sound propagates down the ear canal (ultimately), so I would think testing using a speaker would be more valid than using a headphone....and using a speaker would solve the seating variation issues of headphones that happen above 10kHz.)

You need a very quiet room (like anechoic chamber- quiet) if you used a speaker so that would add a few $100

I certainly don't know how they can create valid results up to 20kHz using headphones when small changes in seating variation creates massive changes to frequency response above 10kHz. But to offer an answer to your post (if I understand correctly how this functions), then the device doesn't measure at the eardrum, it just produces the sounds, and then the person wearing the headphones says whether they can hear it or not - the device records the hearing thresholds, based on the answers from the subject and the person who's conducting the testing & controlling the test tones (which are produced by the machine).

(Re calibration, I would think that would be problematical too, headphones are unnaturally positioned vs speakers so would affect how sound propagates down the ear canal (ultimately), so I would think testing using a speaker would be more valid than using a headphone....and using a speaker would solve the seating variation issues of headphones that happen above 10kHz.)

Yes, audiometers generate the test signal and the measurement is the subjective response: do I hear it or not. The SPL of the tone produced by the audiometer needs to be known so we can calculate the threshold of hearing and amount of hearing loss (HL) based referenced to absolute (o HL) threshold.. I can see using a loudspeaker as being problematic as you need a very quiet room to avoid masking, and tweeters with that can produce very high SPL (over 110 dB SPL ) based on the SPL needed at HF. The speaker will be highly directional at those frequencies so the positioning of the listener (instead of the headphone) becomes the variable... Maybe use some tracking laser to monitor this -- or lock their head in a vice grip.

Here is a paper on the subject of HF audiometry with thresholds of 300+listeners..They use an otoacoustic emissions probe tube in the ear canal that also generates the test signal They are able to measure SPL of the test signal in the ear, estimate the distance of the source to the ear drum based on the first 1/2 wave resonance to calculate what the SPL would be at the ear drum. So it self-calibrates itself for each listener. Clever.

To hear harmonics above 15 kHz in music through headphones you need SPLs in excess of 60-100 dB depending on age and other factors... It's just not going to happen for most of us..

Behavioral Hearing Thresholds Between 0.125 and 20 kHz Using Depth-Compensated Ear Simulator Calibration

The purpose of this study was to obtain behavioral hearing thresholds for frequencies between 0.125 and 20 kHz from a large population between 10 and 65 years old using a clinically feasible calibration method expected to compensate well for variations ...

www.ncbi.nlm.nih.gov

 

[GaryH]

Well, the research shows trained listeners are better able to discriminate better between speakers than reviewers, and for untrained listeners in general for headphones. (There may be a study with reviewers evaluating headphones, but off the top of my head I do not recall one). Asking reviewers to see if they meet the qualifiaction of a trained listener, and if not then to train their ear, I think is reasonable. I am surprised Gary is the first to bring this up, I'll be the second to request this of reviewers. I'll tag @Resolve since he expressed interest in this and what I talk about below may be relevant to him as well.

There also seems to be some confusion regarding what the statistics of trained vs untrained listener rating headphones mean. Passing the program does not mean your subjective impressions are valid and no doubt can be cast upon the - this is an incorrect interpretation of research.

The differences between trained and untrained listeners describe everyone studied, and may not accurately describe an individual. Some untrained listeners will be be equally consistent in blind subjective assessments of headphones or speakers, just on average a random untrained listener is worse than a random trained listener. You may already meet the criteria for being a trained listener, and even if you don't, your subjective opinions if blind tested in some study may still be consistent, similar to that of a trained listener.

Just because someone is a trained listener does not mean their subjective impressions will be reliable. There will always be bias - no subjective impressions will be 100% valid and reliable. The science shows subjective impressions are more reliable when a listener is trained. This does not imply that every subjective impression by a trained listener reliable and valid, just that on average they are more reliable than untrained listeners. Nor does this imply that an impression by an untrained listener is necessarily invalid.

Of course, this is not specific to you. If you do not meet the criteria of trained listener, it does not mean every personal opinion and subjective impression on your website would suddenly become invalid. The science does however show that subjective assessment is more reliable if you are a trained listener. Seeing as this is a science oriented forum, and subjective assessments are a necessary part of reviews, I think it is good to ask reviewers to check if they meet the criteria for being a trained listener, and if they do not, it is good to ask them to train their ear as research participants have done.

Thank you. Finally someone who understands that the reliability and trustworthiness of subjective sound judgements is a continuum, and being a trained listener increases the likelihood of moving higher up that continuum. Why any reviewer would not be interested in doing just that by improving their listening abilities (or at least analysing them) using Harman's How to Listen is frankly baffling. I suspect it may be in part due to some kind of fear that their abilities may not match up to their own estimations, especially for the 'older gentlemen' who are likely to have significant presbycusis by now (as evidenced by the above graphs posted by Sean showing increasing hearing thresholds with age, particularly significantly from around age 55). Either that or they've already taken the test and don't want to reveal the true limits of their abilities, in case they do not match up to their readers' estimations, or at least what they have intimated to their readers (without substantiation) that their 'experience' as a reviewer has done for their abilities. I'll leave this quote on hearing deterioration from @Floyd Toole 's book here (emphasis mine):

These are unfortunate afflictions, especially for audio professionals and product reviewers whose judgments are no longer representative of normal hearing listeners. No amount of experience can compensate for the inability to hear the lowest 20 or 40 dB of musical dynamics, timbral subtleties, distortions, and noises. Seeking opinions of younger ears is always a good idea. An audiogram should perhaps be part of the personal résumé of people in certain sensitive areas of the audio business, displaying evidence of why anyone should trust their opinions about sound quality.

Along with this accompanying figure of his, showing the higher hearing thresholds of listeners who exhibited high variability (poor reliability) in their sound quality ratings:

There also seems to be some confusion regarding what the statistics of trained vs untrained listener rating headphones mean. Passing the program does not mean your subjective impressions are valid and no doubt can be cast upon the - this is an incorrect interpretation of research.

As far as I can see not a single (real) person has claimed that here - only straw men set up by other posters, for some reason in my image

 

[amirm]

There is more to becoming trained listener folks than just taking the Harman How to Listen test. You need to participate in such tests to learn how to evaluate speakers and headphones in that environment and get experience. So I don't see the value of putting pressure on reviewers to do well in that program.

 

[preload]

I am curious why some folks here seem to be so invested in this concept of discrimination btw. And why that seems to be more important, or at least just as important as consistency. Perhaps someone could also try to explain that a little better.

From an experimental research design standpoint, both discrimination and consistency would be important. You can get away with less discriminatory ability (i.e. non-trained listeners) but in order to obtain statistically significant results, you would would require a much larger sample. This, in turn, increases the cost and time required to obtain meaningful results. I seem to remember this was explained in one of Olive's paper, but it also makes intuitive sense.

 

[preload]

Perhaps I'm missing a nuance here, but if the argument is that the typical audio reviewer needs to go through Harman training in order to produce reliable and valid reviews, I completely disagree with that. The problem ain't the hearing, it's the incredible bias and financial influences that permeate this industry.

 

[DualTriode]

Well the obvious answer is that getting a reproducible signal to the eardrum at those small wavelengths is challenging. Small changes in positioning of the headphones on the human ear and coupler etc have huge effects at those frequencies. So I'm sure the threshold tolerances are likely quite large for these reasons and the fact that thresholds are likely quite elevated at those HF frequencies. For in-ear headphones reproducible measurements at HF are challenging and will vary depending on the silicone tip used and whether it blocks the sound when inserted in an ear simulator or in some cases the inside section of the rubber pinna may block the HF.. One reason why we didn't use a pinna (just the coupler with a metal nozzle) when we did our IEC measurements for the research


Likely the HF headphones and the couplers they use for audiometry calibration are designed to minimize these variations in measurement. Removing the pinna from the measurement would be the first step.

Hello All,

Are you missing the irony of this?

We are talking of removing the pinna from an Ear Simulator to isolate and focus on the frequency response of a HF audiometric headphone. (The old version RA0045 was used to develop the Harman Curve)

Next we will be talking about replacing the low frequency GRAS RA0045 coupler with the “the new normal” HF RA0401/02 Ear Simulator.

The next step after that will be to remove all the peaks and dips of the entire Ear Simulator to isolate and focus on the peaks and dips of the audiometric headphone alone.

We will be looking at a flat plate test fixture with a laboratory grade calibrated pressure microphone to study the FR of the audiometric headphones.

This sounds like, “Flow Chart Confusion” to me.

Thanks DT

See:

GRAS RA0401 Externally Polarized High-Frequency Ear Simulator

The GRAS RA0401 is a high-frequency ear simulator with an acoustic input impedance closely resembling that of an average human ear. It includes a 40AG 1/2" pressure microphone and is individually calibrated with this specific microphone.

www.grasacoustics.com

The New Normal

The Standard IEC 60318-4 (711) Ear Simulator (GRAS RA0045) has a step half-wave resonance at 13.5 kHz which effectively limits its usefulness to below at least 10 kHz and it uses a ½" microphone.

However, the RA0401/02 has a damping system that attenuates the half-wave resonance at 13.5 kHz and thus extends the useful frequency range to 20 kHz. It uses the same ½” microphone as the original version and fully complies with the standard. This is why we recommend it as the first choice when you are looking for a “standard” 60318-4 ear simulator - but without its shortcomings at high frequencies. We think of it as “the new normal”.

It is now possible to measure up to 20 kHz. Total Harmonic Distortion (THD), frequency response and driver resonance related phenomena can now be investigated, and an objective supplement to the ”golden ear” approach is now available for concept validation, R&D, and production testing – all based on a ”human-like” test method. In the High-Frequency Ear Simulator, the length-related resonance is damped and the resulting frequency response is much clearer. It is now easier to distinguish the headphone driver resonance from the ear simulator resonance. THD measurements are now less influenced by resonance-related peaks.