• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

ASR Getting Into Measuring Headphones!

OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,370
Likes
234,430
Location
Seattle Area
@amirm

I'd like to see the no-signal noise floor of the measurement rig.
It is pretty high right now because it is next to Audio Precision that is whaling away. Fortunately it doesn't impact the frequency response tests. And for distortion, I am looking at only reporting THD without noise to reduce that impact. I am also looking at a sound proofing box but man, I just don't have the space for it.
 

Helicopter

Major Contributor
Forum Donor
Joined
Aug 13, 2020
Messages
2,693
Likes
3,944
Location
Michigan
Tell you the truth. I don’t care that it’s barely perceivable. I just want to know/have the best „instrument“ out there. I guess a little bit of the same reason why people buy fast cars but rarely if ever even hit half of those speeds which are capable.
That comparison seems a bit unfair for SINAD. Even if performance is beyond what you can hear, you are still getting all that is there. The comparison does work for amp power and speaker power handling.
 

the_brunx

Senior Member
Joined
Apr 7, 2020
Messages
341
Likes
859
It's not that nonlinear distortion levels are barely perceivable, it's that for the large majority of headphones, they're imperceptible. Like the DAC/amp SINAD charts, above the lowest red tier the numbers are purely an engineering metric with no relevance to the end user or sound quality. If we have to do ridiculous audio/car metaphors, then it would be like having a (autonomous) car that can travel faster than any other ever built, but only at top speed when you're asleep so you can never perceive that, and only in a continuous round trip at that speed so it can't take you anywhere faster than a normal car. In other words, pointless.

LOL
I‘d pay to sleep in such a car, for that deep sleep alone. It’s worth it
 

Francis Vaughan

Addicted to Fun and Learning
Forum Donor
Joined
Dec 6, 2018
Messages
933
Likes
4,697
Location
Adelaide Australia
The science shows that nonlinear distortion plays a minimal role in perceived sound quality of headphones e.g. see this paper by Steve Temme and Dr Sean Olive:

It think that is overstating their position a little.
none of the headphone distortion measurements could reliably predict listener preference ratings based on audible distortion
...
this study provides further experimental evidence that traditional nonlinear distortion measurements are not particularly useful at predicting how good or bad high caliber headphone sounds

I think that the key point is that they don't feel that there is a useful traditional metric that allows them to make a predictive model. They absolutely have not said that distortion doesn't matter. Somehow the idea that distortion doesn't matter has evolved as a popular interpretation of these statements. But that isn't what they said. This is why attempting to use the Gedlee or de Santis and Henin metrics would be so useful. These metrics attempt to create a realistic figure of merit based upon human perception. We know from long experience that SINAD by itself does not match human perception and whilst a good way of determining that an amp or DAC (say) embodies inaudible distortion, when the distortion is audible, it fails to allow us to decide what sounds good or bad. Two devices with wildly different harmonic distortion spectra can have the same SINAD, but sound quite different, and one can sound good and the other bad. Trivially high order odd harmonics sound vastly worse than much higher levels of even low order harmonics.
They tried THD, IMD, Multitone and Non-coherent distortion measures. But none of these take into account human hearing mechanisms. They don't take into account even the basics of masking effects. Which make them pretty useless.

So as Temme and Olive say "traditional nonlinear distortion measurements are not particularly useful". Nobody disagrees. So lets try a vastly more modern and scientifically based metric or two.
 

DualTriode

Addicted to Fun and Learning
Joined
Oct 24, 2019
Messages
893
Likes
593
hello All,

Geddes tells us that H2 , H3 and IM distortions are masked by the program material up into double digit % distortion, higher order; H4, H5 and above is another story. The greater the SPL the greater the masking.

Thanks DT
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,370
Likes
234,430
Location
Seattle Area
Electromechanical devices are complex beasts, trading one kind of distortion for another depending on frequency, level or both. I have been working on refining my tests and finally getting some place with a graph that makes sense. Once I measure more headphones I can qualify it better.

On possible advantage of low distortion is how much you can boost the response especially in bass. In some cases we are talking about 5 to 10 dB increase so non-trivial amount with response to what the driver is doing.
 

RayDunzl

Grand Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
13,201
Likes
16,983
Location
Riverview FL
Crude Experiment:

27.5Hz fundamental in room both speakers no subs. 12" sealed woofers.

1607417459488.png




Approximate thresholds for me hearing the harmonics:

Harmonics added one at a time, but not listened for individually.

1607417877432.png


As music, I'm sure the thresholds would be much higher.
 
Last edited:

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,875
Likes
6,673
Location
UK
Electromechanical devices are complex beasts, trading one kind of distortion for another depending on frequency, level or both. I have been working on refining my tests and finally getting some place with a graph that makes sense. Once I measure more headphones I can qualify it better.

On possible advantage of low distortion is how much you can boost the response especially in bass. In some cases we are talking about 5 to 10 dB increase so non-trivial amount with response to what the driver is doing.
That's a very good practical point, I just ordered a Drop + Hifiman HE4XX because I wanted to try the planar experience, they get good soundstage ratings, they have an EQ'able frequency response as measured by Oratory, and bass on planars is supposed to be fantastic due to low distortion....but it would mean EQ'ing up the bass by 8dB for 30Hz and 9dB for 20Hz to get it bang onto the Harman 2018 curve all the way down to 20Hz, so I'm hoping it will be able to handle that without problematical distortion - so distortion measurements I see as being very applicable. For instance, my experience with EQing up the bass on the Sennheiser HD600 can make it sound quite flabby & undefined (in contrast to my closed back NAD HP50 benchmark), which I'm thinking is related to the relatively high level of measured distortion that HD600 has in it's bass. Any kind of advancements in distortion measurements from this site would be worthy in my opinion.

EDIT: for curiosity this is the EQ I created just now (pics at end of post) for the Drop Hifiman HE4xx which is based on Oratory's measurement, and is done to the same Harman 2018 Curve, but mine is more accurate to the curve in the bass & treble as well as I predict not being as harsh as Oratory's in the +10kHz area as he used a High Shelf filter to economically boost dips from 1kHz upwards but he didn't ever cancel out the High Shelf boost north of 10kHz therefore his is +6dB boosted across the whole 10kHz-20kHz range which I think is a little crazy, which you can see from this pic of his EQ result (Hifiman HE4xx Oratory's EQ result.jpg). I chose to EQ up the 5450Mhz dip more than he did, but I did end up using lower Q filters than he did, but when I get the headphone I might choose not to boost up that dip quite as much as I have. On the subject of distortion this EQ of mine required a negative preamp of -8.5dB due to a +8.5dB boost of the bass, but that's pretty much bang on the Harman 2018 Curve down to 20Hz - would be interesting to see what kind of distortion this would produce on this particular headphone, perhaps Amir will measure this headphone on the Gras like he did on the B&K. Here's my EQ I did pictured with & without the filter listing (the filter listing gets in the way of seeing the frequency response properly....I used 12 filters for this EQ, ignore the ones that are greyed out as they are not active & were just part of my experimentation/creation process:
Hifiman HE4xx My EQ.jpgHifiman HE4xx My EQ (without filters displayed).jpg
 
Last edited:

ferrellms

Active Member
Joined
Mar 24, 2019
Messages
296
Likes
254
The heading says it all: we are going to be officially measuring, reviewing and characterizing earphones and headphones (likely more of the latter than the former). Some of you know the evaluation process I went through this summer of the BK 5128 Head and Torso headphone measurement system. The company was wonderful to work with but alas, lack of research using that new system made interpreting the results very difficult. And then there was the cost for which you could buy a luxury car! Post that I evaluated the GRAS 45CA which has been used in research and results were much more encouraging and the cost much lower. So a couple of months ago I decided to put in the order for one and here it is!

View attachment 97136

The fixture is incredibly heavy but that is goodness as it won't move when you try to adjust the headphone. Because the fixture is not a manikin, I actually find it a lot easier to try different positions to get better alignment of the headphone. The unit is stereo so I am able to quickly compare one channel to the other for alignment purposes and also for driver matching.

There are many variations of 45CA. The one I have purchases is the 45CA-10. Options on it are anthropometric pinna (human looking ear), model KB5010, which you can see more clearly here:

View attachment 97138

The old pinna were rigid unlike a real ear. That caused headphones that sat on the ear to sit proud, completely messing up their low frequency response. The pinna as supplied is quite soft -- much like a babies ear. My own ears are more rigid (as they should be for someone as King of Audio!!!). So some variation always remains but at least we are closer.

The other option is the high-frequency ear simulator/coupler (microphone) model RA0402. Traditional couplers were of "711" type and had too much resonance in higher frequencies. The RA0402 has a more muted response in that region. This would avoid the temptation to try to pull those peaks down post measurements as they were likely not matching real ear response. That said, the correction is somewhat arbitrary to so confidence remains low in higher treble region (due to this and reflections inside the cup).

External constant current is needed to drive the microphone pre-amplifiers built into the system. So I had to purchase a GRAS 12AX 4-channel power supply and amplifier (though I will be using it as a 0 dB buffer). The microphones are pre-polarized meaning you don't need a high voltage power supply to drive then as you used to.

The retail cost is around $15,000 so this was another significant investment in our toolset to quantify performance of audio products.

I know some of you think headphone measurements have already been done and other think they are inaccurate so whey bother. I had to resolve both of these issues in my mind before deciding to spend this money so let me give you my justification for proceeding.

1. Measurements have been done. This is indeed true for the most part. It is impressive how the headphone community has embraced measurements to such a degree compared to speakers which was hardly touched as far as proper measurements. Part of this is due to lower cost of measurement gear but still, the effort to measure literally hundreds of not thousands of headphones was not free.

While the data is great, having so many versions of it is not so. There are a number of different fixtures uses resulting in different measurements. Super experienced headphone users know how to wade through these and get the information they need. But it is not for everyone.

Then there is our method of doing things. You could have found electronic and speaker measurements elsewhere but we have managed to not only catch up but leave behind many other sites that do this type of work. There is a "style," method and even attitude that goes into my reviews that is unique and works for both me and the readership. This combination does not exist elsewhere as it is unique by nature of it. Indeed the few headphone reviews I have done created a ton of interest. Importantly for me, they quantified the performance of the headphones I tested beyond what I had read elsewhere. Maybe that is my issue but again, how to clearly communicate what a piece of audio is about is what I specialize in. And will bring the same to headphones.

2. Why bother to measure headphones as data is inaccurate. This is what led me last year to invest in speaker measurements than headphones. The research about speakers is far more conclusive and measurements a ton more standardized. But something changed when I got the GRA 45C evaluation system. I measured, saw the deficiencies, applied judicious EQ that resulted in incredible sonic improvement in these headphones. This demonstrated that the objective data is indeed highly instructive. And that just a bit of EQ transforms the performance of any headphone, making them not only more accurate but far more enjoyable. As I type this, I am listening to AEON Flow closed headphone post EQ and it sounds so nice. In some respects, it is outperforming some of the best speakers I have tested!

Yes, you have to be careful how you approach the measurement data. I am not a fan of hugging a target curve to the max and spitting out 10 EQ bands to apply to them. The measurement data is not that reliable to enable this. But a softer approach works and works wonderfully.

A bonus of measuring headphones relative to speakers is speed. I am able to characterize the frequency response of a headphone in 5 seconds. Same process takes hours with speakers and lugging around heavy boxes around.

Ultimately I thought I had no choice but to test and review headphones. They are part and parcel of my own music listening and countless audiophiles. We have to treat them as well as we do other audio products, Leaving their testing to others did not make sense leaving a hole in our arsenal of bringing the best sound to everyone.

The main barrier was of course the cost. The capital investments in this effort have long gone the crazy land even when I purchased my Audio Precision analyzer let alone the Klippel speaker measurement system. I don't know when I am going to recoup those dollars. So investing even more did not at all seem logical. But I just decided to hell with it and proceed. Asked my wife what she thought and she could tell in my eyes there was not a "no" in what I was expecting from her. :)

As with our other measurements, there will be some growing pains as we develop and mature our measurement suite for headphones. I will as always carry a hard line in listening to complaints. :) But I will internalize some and will make them more perfect as time goes. I ask you to leave negativity behind and work constructively to get more data under our belt about this super important transducer. Last thing I need is dealing with food fights on this subject in addition to new workload of testing headphones.

So there it is. I am poorer financially but am confident I will be far richer emotionally to bring more data to our collective audio life.

P.S. I picked up the 45CA in person from Audio Precision headquarters since they are the sister company to GRAS. Alas, half way down the check engine line came on in our Mercedes Sprinter RV. :( Read the code and it is two emissions (NOx) sensors going bad. Hear huge horror stories about how expensive such repairs are at Mercedes. Have a call into them to see if they are covered under emissions warranty (the van is 4.5 years old with only 18,000 miles). I got home only to see a message from them that the power supply had just arrived and was not in the box I picked up! So I had to wait a week for that to arrive. Not a good start but hopefully things will change from here on.

------------

Any donations are appreciated using: https://www.audiosciencereview.com/forum/index.php?threads/how-to-support-audio-science-review.8150/
Your speaker measurements are unique and especially valuable. But why should I donate to an effort that simply duplicates something that has been done many times before? One-off measurements of headphones are everywhere on the web. Of course, unit to unit variance is not captured, and it most definitely exists.

Sonarworks makes its money by selling eq filters for headphones (and room correction, too). You can have them custom-made if you send the phones to them, or you can purchase filters for the average response of your phone's model that they have measured in creating their custom filters for that model. They have thousands of happy customers around the world, and are particularly used by recording and mastering engineers.

I am not sure if you have to be a customer to get access to the database of measurements, but as a happy customer I will trust their averaged measurements over your one-offs.
 
Last edited:

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,875
Likes
6,673
Location
UK
Your speaker measurements are unique and especially valuable. But why should I donate to an effort that simply duplicates something that has been done many times before? One-off measurements of headphones are everywhere on the web. Of course, unit to unit variance is not captured, and it most definitely exists.

Sonarworks makes its money by selling eq filters for headphones (and room correction, too). You can have them custom-made if you send the phones to them, or you can purchase filters for the average response of your phone's model that they have measured in creating their custom filters for that model. They have thousands of happy customers around the world, and are particularly used by recording and mastering engineers.

I am not sure if you have to be a customer to get access to the database of measurements, but as a happy customer I will trust their averaged measurements over your one-offs.
I'm sure I've heard bad things about Sonarworks re headphones, but can't remember even vague specifics, given that I hardly ever hear about them I'd be surprised if they're exceedingly valid in what they do with headphones....because I think I'd know a lot more about them otherwise. But I'll have to let others who know specifically what equipment they measure on & what curve they EQ to can answer the specifics. Sure I've heard bad things about them though. (Yes, I know, I'm shooting from the hip on this point!)

Amir's work is not a duplication, it's an addition, and perhaps there can be some unique takes on it & some insights not achieved elsewhere - I think this is a perfect site for contributing to the validity of headphone measurements & science, given it's ethos & integrity.
 
Last edited:
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,370
Likes
234,430
Location
Seattle Area
Your speaker measurements are unique and especially valuable. But why should I donate to an effort that simply duplicates something that has been done many times before? One-off measurements of headphones are everywhere on the web. Of course, unit to unit variance is not captured, and it most definitely exists.
Show me one, one, measurement out there that has a pink panther rating. You won't find one! :D

As to donation, people donate when they feel good about what we are doing here. And what we are doing has to encompass the full fidelity chain of audio. We can now go from electronics to speakers. There was no choice but also travel that chain to the headphone.

Sonarworks makes its money by selling eq filters for headphones (and room correction, too). You can have them custom-made if you send the phones to them, or you can purchase filters for the average response of your phone's model that they have measured in creating their custom filters for that model. They have thousands of happy customers around the world, and are particularly used by recording and mastering engineers.

I am not sure if you have to be a customer to get access to the database of measurements, but as a happy customer I will trust their averaged measurements over your one-offs.
I am familiar with the company and downloaded a trial version of their software. Maybe one of the projects we do is compare their EQ for a few headphones to measurements we make. We can analyze what they have done and see if it is or is not proper and whether better EQ exists, or not. You have seen anyone do this?

Another more immediate project that is pending is whether tube Stax amps result in different performance in electrostatic headphones than solid state. I have done this test electrically but plan to do it post headphone reproduction. Seen anyone do this?

Quantifying distortion is of interest. Most measurements posted for headphones are open-loop with a single distortion snapshot or maybe two. My measurement system is closed loop meaning I can test many levels and report in a single graph. We could add EQ and then see how distortion changes. Have you seen this elsewhere?

As they say the night is young. More fresh ideas will come out when the combined energy and thinking of membership is combined with the tools and time I have to develop unique insight into speakers. The other measurements are results of the initiator.

When all is done, it may not provide enough value for you. But I know it has already for me, prompting me to spend significant amount of money. And I had read and seen all the measurements you have seen. This may be an indication of me being too dense to appreciate what is out there. :) But either way, I am seeing results that I value highly. If you don't and you don't donate, that is cool. I purchased the tool without expectation of people automatically paying for it.
 

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,875
Likes
6,673
Location
UK
Quantifying distortion is of interest. Most measurements posted for headphones are open-loop with a single distortion snapshot or maybe two. My measurement system is closed loop meaning I can test many levels and report in a single graph. We could add EQ and then see how distortion changes. Have you seen this elsewhere?
Wow, I really like this idea! This is right up my street, very applicable.
 

bobbooo

Major Contributor
Joined
Aug 30, 2019
Messages
1,479
Likes
2,079
Wow, I really like this idea! This is right up my street, very applicable.

As EQ is just an amplitude change at a given frequency, it will likely just increase (or decrease) the absolute distortion as if the SPL (at each given frequency) was increased to that amplitude, but it would be good to check this. More importantly though, measuring post-EQ would show up any errors in the frequency response that are not able to be EQed out effectively due to non-minimum phase behaviour, by comparing this post-EQ measurement with the pre-EQ measurement convolved with the EQ curve, and seeing where they differ (if at all).
 
Last edited:

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,875
Likes
6,673
Location
UK
As EQ is just an amplitude change at a given frequency, it will likely just increase (or decrease) the absolute distortion by that amplitude (at that given frequency), but it would be good to check this. More importantly though, measuring post-EQ would show up any errors in the frequency response that are not able to be EQed out effectively due to non-minimum phase behaviour, by comparing this post-EQ measurement with the pre-EQ measurement convolved with the EQ curve.
Yeah, sounds good, it's worth doing to validate that the EQ is actually "taking" - your minimum phase comments. Probably quite a bit of extra work due to all the many extra measurements you'd need to take to compare averages between the two.....so might be best to only do this for headphones that look like they have non-minimum phase issues in the frequency response.

I'd still like to see the distortion measurements post EQ, perhaps with bass boosted up to Harman 2018 curve and the headphone played at a normal to loud listening level....as this would be very applicable to the headphone model being assessed, you could easily see with confidence how far you could EQ the bass....perhaps it could even be worked out how much you could increase the bass until distortion passes a certain threshold, which would be different for each headphone (whilst at normal to loud playback level)....but it could be an interesting & applicable way of characterising a headphones ability to have it's bass EQ'd to whatever taste (all the way up to extreme bass head).
 
Last edited:

bobbooo

Major Contributor
Joined
Aug 30, 2019
Messages
1,479
Likes
2,079
I reckon that headphone that got the highest preference rating could be the HifiMan HE400S - it's listed as $300 which matches up exactly with the data point. Similar frequency response to the HE4XX (with the latter actually having better sub-bass response).
 
Last edited:

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,554
Likes
1,701
Location
California
I reckon that headphone that got the highest preference rating could be the HifiMan HE400S - it's listed as $300 which matches up exactly with the data point. Similar frequency response to the HE4XX (with the latter actually having better sub-bass response).

Maybe, but there are 7 models between $295 and $340.
 

DualTriode

Addicted to Fun and Learning
Joined
Oct 24, 2019
Messages
893
Likes
593
Hello All,

Is this open loop testing or closed loop testing? Open loop testing uses an outside recording or playback device. Closed loop testing goes from the generator possibly to amplifier to DUT to sensor to analyzer. The way that I have headphone testing set up on my bench I think of it as a broken loop leaning towards open loop. There is a lot of outside the loop instrument and fan noise.

I ran a couple of tests first with the APx1701 test amplifier turned off and the second time around with the amplifiers turned on.

The first test had no test signal as output coming out of my old favorite HD600’s there was plenty of exterior fan noise that showed up on the plots as distortion.

The second test with the amplifiers turned on gave us the usual HD600 Frequency Response plus a THD distortion plot that looked very much like the first THD distortion plot that was only fan noise.

For more information the fan noise, 52dB SPL, in an otherwise much quieter space.

Thanks DT
HD600 THD Level amplifiers off.PNG
HD600 THD Level amplifiers on.PNG
HD600 RMS Level amplifiers off.PNG
HD600 RMS Level amplifiers on.png
 
Top Bottom