• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

HBK Headphone Measurement Talks from Head-Fi and Sean Olive

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
Bruel & Kjaer (BK), makers of precision acoustic measurement products, had a nice two day virtual conference. I wanted to listen to a bunch of them but had a major conflict. Fortunately, I watched two talks related to headphone measurements:

First one was by Jude Mansilla of Head-fi with the curious title of, "Audio Measurements As Consumers Content."

The second was from Dr. Sean Olive of Harman, going through his standard presentation on headphone preference but also a new research addition: difference between various headphone measurement gear relative to the one they used in their research. Specifically we were treated to the first analysis of BK 5128 which Head-fi has adopted as their standard.

Jude Presentation
I was very surprised and disappointed in most of this presentation. Bulk of time was spent showing and ridiculing the DIY measurement rigs members of his own forum had created. He showed pictures of them and while correctly stating some issues with them, I just could not figure out what he is after. Hobbyist are creating these measurements because headphone companies are not providing them. If Jude wanted to improve things, he should have complained about lack of such measurements from that sector which would be likely to be in this conference, than DIY people. I personally admired the work and creativity people had put in building their various fixtures.

Fortunately the second half of the talk was better in that he showed a bit about how they measure headphones which seems to be following what Tyll did with use of square wave and such. I am personally not a fan of driving headphone or speakers with square wave, especially a low frequency one. This can be hard on the transducer with the long duty cycle essentially being DC. I saw little justification for this method other than a hack to show the frequency response, sort of, using crude FFT.

The other thing he mentioned is that they no longer calibrate at one frequency and instead use white noise. I don't understand the merit of this either as matching the measurements to target needs to be done in a way that relates to the research. Credit to him he asked for feedback from audience but none was provided. Target matching is a visual thing for humans anyway so ultimately it doesn't matter per se.

I finally got an explanation of why they spent so much money on the Herzan isolation chamber. I am able to make measurements with better noise floor than them in my office with no isolation chamber. Answer is that they work in a noisy and busy office building and without noise isolation, they would have fair bit of pollution. In their typical promotion, they always made it sound like this was an essential thing in any measurement system rather than being a requirement in their noisy environment. We are fortunate to live far from civilization so have the quietness most of the time sans the delivery truck and our dogs barking because of that. I do my acoustic testing at night which eliminated this issue. Anyway, having an isolation chamber is not a bad idea but context is important and was not stated until now.

In QA section he was asked what his favorite headphone was. He said the Sennheiser HE-1 ($45,000). He was asked if he had measured it. Shockingly he said no! Gosh that was awkward when you are in a measurement seminar and you don't believe in this stuff to practice it.

Along these lines, he was asked if he felt listening tests were needed in headphones once you have measurements. He said no but could not provide any reasoning why.

Dr. Olive Presentation
As I noted, the meat of the presentation was a new research project to go back and remeasure a 20 headphones against half a dozen HATS and measurement gear. They only disclosed the difference between the modified GRAS 45 that they had used and BK 5128 which happened to be the one I, we are all interested in. The results were fascinating.

The BK 5128 underreported bass frequencies by substantial amount and in a sloping way. The lower the frequency, the less it reported the bass energy. This caused the preference score of a headphone drop 45 points from high 80s down to 40s! There were also some differences in high frequencies but that was not that meaningful or emphasized in the presentation.

They developed a compensation curve for 5128 relative to Harman target but alas, not all headphones showed the same differential. Using this new target, the above headphone showed an error of I think 6 points. But there are others that cannot be fixed this way.

To summarize, there is no way to use the Harman target curve 100% reliably when performing measurements using BK 5128! You can get close for majority of cases if you compensate but not all. And there could be effect on headphones that were not measured in the study.

So the overlaying that Jude does at head-fi with Harman target is totally wrong and inappropriate for bass energy.

As a proof point, Sean actually pulled my measurements of the Dan Clark Stealth and compared it to Jude's:

Mine:
index.php


Judes:

1633039279604.png


He matched the sizes and played with the reference levels to get an apple vs apple comparison. It clearly showed deficiency in bass measurements which he could explain with their research. But Jude's also showed a deficiency in the 3 to 6 kHz which I have circled in read which Sean and team could not explain. The dip does not show up in my measurements. That seems to be a measurement error in Jude's measurement but of unknown cause right now.

This is a fairly big setback for Head-fi headphone measurements based on BK 5128. No benefit was identified in using it, and problems identified with what it does produce. Fortunately some of it can be fixed and I hope Jude adopts the modified curve as to reduce confusion that his measurements can create relative to Harman research and comparison with measurements that the rest of us are performing.

I have been making this point for quite a while that more precision in some measurement doesn't do you any good if you don't have a yardstick to compare it to. And that seems to be the case with BK5128. I think fair bit of the research money in development of that fixture should have gone toward creating realistic target curves for it. Without it, the hype has gotten ahead of its true value putting head-fi in a difficult situation right now.

I felt bad for Jude as Sean's presentation was right after his. He had just finished singing the praises of BK5128 but all he showed for a proof point was the same IEM measurements he has been showing and how an IEM they tested was bright and GRAS measurement didn't show it but BK5128 did (or the reverse, can't recall exactly which).

Sean was also asked about his favorite headphones and he mentioned a couple that were in under $200 range.

When he covered the difference bass preferences, he was asked what they target. He said that the JBL line is boosting the bass a couple of dBs with the assumption that younger crowd were buying so were more inclined to want more bass. AKG line was staying true to the target bass response.

One sour note was Samsung Legal censoring one of his slides. And the fact that the company's policy has become to make sure competitors are NOT taught how to compete with Harman as they disclose their research. This means we will not hear as much as we have heard in the past about their work. I can see why Samsung has taken this position from business point of view, but for the rest of us, it is not a good development.

I hope Samsung management/Legal understand that by sharing their research, their work has become a review criteria which can be very beneficial to them.

That's all I remember to post. I have asked HBK if they will provide on-demand versions of these presentations but I have not heard back.
 
Last edited:

thewas

Master Contributor
Forum Donor
Joined
Jan 15, 2020
Messages
6,739
Likes
16,161
The other thing he mentioned is that they no longer calibrate at one frequency and instead use white noise.
Which is also weird, at least for better (psycho)acoustic loudness similarity pink noise would be a better choice.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
I feel that naming a $45,000 headphone was a total copout that he is wagering he will not be questioned on simply because no one can afford to measure it or even use their ears to evaluate his completely subjective opinion.
It certainly came across the typical, "it is so expensive of course it was good."
 

thewas

Master Contributor
Forum Donor
Joined
Jan 15, 2020
Messages
6,739
Likes
16,161
What would be much more important to see is how much the delta of different headphones measured with the 2 rigs varies, as if it does significantly a static "target conversion" wouldn't really work.
 

staticV3

Master Contributor
Joined
Aug 29, 2019
Messages
7,531
Likes
11,997
One sour note was Samsung Legal censoring one of his slides.
(Un)fortunately censoring just the target while still showing raw FR and error curve based on the new target is darn near useless.
Even the super pixelated screenshot shared on twitter is enough to reconstruct the "hidden" B&K 5128 Harman target 50-10,000Hz with ±0.25dB accuracy. Maybe even less with smart smoothing. 20-50Hz is easy enough to estimate based on fixture response delta, which leaves only 10K-20K as a Samsung trade secret.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
From what I recall, it was a small percentage that didn't work with static correction. The problem is, when you measure a new headphone, how do you know you will hit that difference or not?
 

thewas

Master Contributor
Forum Donor
Joined
Jan 15, 2020
Messages
6,739
Likes
16,161
From what I recall, it was a small percentage that didn't work with static correction. The problem is, when you measure a new headphone, how do you know you will hit that difference or not?
That's still one of the biggest problems of headphones and in the end relates also to the HRTFs of the individual listeners. I would chose the rig which is closer to mine or optimally an average of the population acoustic impedance and transfer functions.
 

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
329
Likes
3,004
Bruel & Kjaer (BK), makers of precision acoustic measurement products, had a nice two day virtual conference. I wanted to listen to a bunch of them but had a major conflict. Fortunately, I watched two talks related to headphone measurements:

First one was by Jude Mansilla of Head-fi with the curious title of, "Audio Measurements As Consumers Content."

The second was from Dr. Sean Olive of Harman, going through his standard presentation on headphone preference but also a new research addition: difference between various headphone measurement gear relative to the one they used in their research. Specifically we were treated to the first analysis of BK 5128 which Head-fi has adopted as their standard.

Jude Presentation
I was very surprised and disappointed in most of this presentation. Bulk of time was spent showing and ridiculing the DIY measurement rigs members of his own forum had created. He showed pictures of them and while correctly stating some issues with them, I just could not figure out what he is after. Hobbyist are creating these measurements because headphone companies are not providing them. If Jude wanted to improve things, he should have complained about lack of such measurements from that sector which would be likely to be in this conference, than DIY people. I personally admired the work and creativity people had put in building their various fixtures.

Fortunately the second half of the talk was better in that he showed a bit about how they measure headphones which seems to be following what Tyll did with use of square wave and such. I am personally of fan of driving headphone or speakers with square wave, especially a low frequency one. This can be hard on the transducer with the long duty cycle essentially being DC. I saw little justification for this method other than a hack to show the frequency response, sort of, using crude FFT.

The other thing he mentioned is that they no longer calibrate at one frequency and instead use white noise. I don't understand the merit of this either as matching the measurements to target needs to be done in a way that relates to the research. Credit to him he asked for feedback from audience but none was provided. Target matching is a visual thing for humans anyway so ultimately it doesn't matter per se.

I finally got an explanation of why they spent so much money on the Herzan isolation chamber. I am able to make measurements with better noise floor than them in my office with no isolation chamber. Answer is that they work in a noisy and busy office building and without noise isolation, they would have fair bit of pollution. In their typical promotion, they always made it sound like this was an essential thing in any measurement system rather than being a requirement in their noisy environment. We are fortunate to live far from civilization so have the quietness most of the time sans the delivery truck and our dogs barking because of that. I do my acoustic testing at night which eliminated this issue. Anyway, having an isolation chamber is not a bad idea but context is important and was not stated until now.

In QA section he was asked what his favorite headphone was. He said the Sennheiser HE-1 ($45,000). He was asked if he had measured it. Shockingly he said no! Gosh that was awkward when you are in a measurement seminar and you don't believe in this stuff to practice it.

Along these lines, he was asked if he felt listening tests were needed in headphones once you have measurements. He said no but could not provide any reasoning why.

Dr. Olive Presentation
As I noted, the meat of the presentation was a new research project to go back and remeasure a 20 headphones against half a dozen HATS and measurement gear. They only disclosed the difference between the modified GRAS 45 that they had used and BK 5128 which happened to be the one I, we are all interested in. The results were fascinating.

The BK 5128 underreported bass frequencies by substantial amount and in a sloping way. The lower the frequency, the less it reported the bass energy. This caused the preference score of a headphone drop 45 points from high 80s down to 40s! There were also some differences in high frequencies but that was not that meaningful or emphasized in the presentation.

They developed a compensation curve for 5128 relative to Harman target but alas, not all headphones showed the same differential. Using this new target, the above headphone showed an error of I think 6 points. But there are others that cannot be fixed this way.

To summarize, there is no way to use the Harman target curve 100% reliably when performing measurements using BK 5128! You can get close for majority of cases if you compensate but not all. And there could be effect on headphones that were not measured in the study.

So the overlaying that Jude does at head-fi with Harman target is totally wrong and inappropriate for bass energy.

As a proof point, Sean actually pulled my measurements of the Dan Clark Stealth and compared it to Jude's:

Mine:
index.php


Judes:

View attachment 156436

He matched the sizes and played with the reference levels to get an apple vs apple comparison. It clearly showed deficiency in bass measurements which he could explain with their research. But Jude's also showed a deficiency in the 3 to 6 kHz which I have circled in read which Sean and team could not explain. The dip does not show up in my measurements. That seems to be a measurement error in Jude's measurement but of unknown cause right now.

This is a fairly big setback for Head-fi headphone measurements based on BK 5128. No benefit was identified in using it, and problems identified with what it does produce. Fortunately some of it can be fixed and I hope Jude adopts the modified curve as to reduce confusion that his measurements can create relative to Harman research and comparison with measurements that the rest of us are performing.

I have been making this point for quite a while that more precision in some measurement doesn't do you any good if you don't have a yardstick to compare it to. And that seems to be the case with BK5128. I think fair bit of the research money in development of that fixture should have gone toward creating realistic target curves for it. Without it, the hype has gotten ahead of its true value putting head-fi in a difficult situation right now.

I felt bad for Jude as Sean's presentation was right after his. He had just finished singing the praises of BK5128 but all he showed for a proof point was the same IEM measurements he has been showing and how an IEM they tested was bright and GRAS measurement didn't show it but BK5128 did (or the reverse, can't recall exactly which).

Sean was also asked about his favorite headphones and he mentioned a couple that were in under $200 range.

When he covered the difference bass preferences, he was asked what they target. He said that the JBL line is boosting the bass a couple of dBs with the assumption that younger crowd were buying so were more inclined to want more bass. AKG line was staying true to the target bass response.

One sour note was Samsung Legal censoring one of his slides. And the fact that the company's policy has become to make sure competitors are NOT taught how to compete with Harman as they disclose their research. This means we will not hear as much as we have heard in the past about their work. I can see why Samsung has taken this position from business point of view, but for the rest of us, it is not a good development.

I hope Samsung management/Legal understand that by sharing their research, their work has become a review criteria which can be very beneficial to them.

That's all I remember to post. I have asked HBK if they will provide on-demand versions of these presentations but I have not heard back.
I posted that comparison on Twitter today here but I've pasted it below. You will find a couple more slides from my presentation on twitter.

The decision to no longer publish research considered useful to the industry is coming from HARMAN management - not Samsung.I've had a lot of scientific/academic freedom and publishing research over my career, and it was always justified as "thought leadership". There was this perhaps naive notion that publishing scientific evidence about what makes products sound good was more effective than just having sales and marketing making unsubstantiated claims that they sound good. But the decision was made that showing scientific evidence also helps competitors follow your lead.

Jude mentioned in his talk that Harman research is the best we have at the moment because we are the only company that shares it. He inferred other companies are doing similar research but being secret about it. . So we will have to wait to see who will be the new thought leader in the industry.



1633046692949.png
 
Last edited:

GWolfman

Addicted to Fun and Learning
Forum Donor
Joined
Nov 3, 2020
Messages
624
Likes
1,041
Thanks for the review of the talks.

The guy mentioning the elite expensive cans as his favorite lost all credibility in 10 seconds, IMHO.
 

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
329
Likes
3,004
What would be much more important to see is how much the delta of different headphones measured with the 2 rigs varies, as if it does significantly a static "target conversion" wouldn't really work.
I have that data. We made 5 reseats for each of the 20 headphones we measured using different amount of pressure on the cups. So a total of 20 headphones x 5 reseats x 3 pressures x 6 fixtures =1800 measurements
 
Top Bottom