• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

HBK Headphone Measurement Talks from Head-Fi and Sean Olive

joentell

Active Member
Reviewer
Joined
Feb 4, 2020
Messages
237
Likes
766
Location
Los Angeles
Only one (potential) taker so far in @Resolve ? Ok I'll open this up to not just headphone reviewers - @hardisj , @Gene DellaSala , @joentell , @napilopez , @John Atkinson , @Kal Rubinson , willing to give it a try?
I like the idea of learning to listen and training.

I think hearing capabilities play a role as does our ability to accurately specify what we are hearing.

Yet another factor is be able to separate our personal preferences/biases from what's objective. How good are we at self-assessment?

Even after all of that, it is still difficult to communicate one's experience into words. In some ways, objective data is easier because you can just present it, and if you understand what you see, then good. But that's almost never the case. Some people will need help understanding what the data means and how it will translate to a real-world experience. That's where some subjectivity can also come into play when one has to pick and choose which data to focus on and which information carries more weight than what.

In the end, we are humans, not robots. I wouldn't "expect" someone to take my subjective opinions as a "useful data point". I would just think they would see it as a subjective opinion the way I see your suggestion as just a suggestion and nothing more. The subjective weight they would give to my opinion would be based on their trust in me, which is also subjective.

The only way that could be measured objectively might be to see how accurately I've assessed products in the past, but even that requires some translation. I would give myself an 80% accuracy score maybe. B grade. I watched an old review of mine today and thought I was too easy on the product, and would've said something different now. I would give myself an A for effort though. ;-) I'm trying my best.

I also want people to know that the expectations viewers have for us is not the expectations we set for ourselves. I started my channel just to "Show & Tell" stuff I thought was cool. I didn't call myself "New Consumer Reports" or anything like that. I don't know that it's fair to expect all reviewers to be like Audio Science Review. That's why I'm happy this place is here, and Amir is doing what he does. I understand there's some responsibility that goes along with the growth of my channel, but that doesn't mean I should have to change the entire purpose of my channel because of relative success.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
I like the idea of learning to listen and training.

I think hearing capabilities play a role as does our ability to accurately specify what we are hearing.

Yet another factor is be able to separate our personal preferences/biases from what's objective. How good are we at self-assessment?

Even after all of that, it is still difficult to communicate one's experience into words. In some ways, objective data is easier because you can just present it, and if you understand what you see, then good. But that's almost never the case. Some people will need help understanding what the data means and how it will translate to a real-world experience. That's where some subjectivity can also come into play when one has to pick and choose which data to focus on and which information carries more weight than what.

In the end, we are humans, not robots. I wouldn't "expect" someone to take my subjective opinions as a "useful data point". I would just think they would see it as a subjective opinion the way I see your suggestion as just a suggestion and nothing more. The subjective weight they would give to my opinion would be based on their trust in me, which is also subjective.

The only way that could be measured objectively might be to see how accurately I've assessed products in the past, but even that requires some translation. I would give myself an 80% accuracy score maybe. B grade. I watched an old review of mine today and thought I was too easy on the product, and would've said something different now. I would give myself an A for effort though. ;-) I'm trying my best.

I also want people to know that the expectations viewers have for us is not the expectations we set for ourselves. I started my channel just to "Show & Tell" stuff I thought was cool. I didn't call myself "New Consumer Reports" or anything like that. I don't know that it's fair to expect all reviewers to be like Audio Science Review. That's why I'm happy this place is here, and Amir is doing what he does. I understand there's some responsibility that goes along with the growth of my channel, but that doesn't mean I should have to change the entire purpose of my channel because of relative success.
Your comments probably underlie a lot of what reviewers are thinking. So I'll answer in general.

Look at Amir: the panthers, which started as a funny, lighthearted thing, are now called "panther ratings" by the forum members. People give them numerical value and analyze them. Reviewers, particularly successful reviewers, aren't the same brand of human as everyone else. Their opinions carry a very different sort of weight.

I don't think you have to adopt a rigorous, measurement based show and review process. Your sincerity is not really in question, but your judgments are. And the reliability of your advice and commentary. So to be consistent with your approach, I think your main responsibility is to acknowledge your level of influence. You might be a regular dude but what you say really counts. Help people understand what your opinions are and aren't.
 

GaryH

Major Contributor
Joined
May 12, 2021
Messages
1,348
Likes
1,804
If you had mentioned this graph was for IEMs to begin with though, then I probably wouldn't have bothered to respond on it. Because imho IEMs are much a less reliable tool for analyzing and predicting users' frequency response preferences than over-ear headphones (or speakers). So I'm not sure why it would be that relevant in this discussion. (?)
Well your opinion on IEMs is contrary to Harman's research, which found a 0.91 correlation between actual and predicted ratings given by listeners in blind tests, compared to 0.86 for both over-ear headphones and speakers. Anyway, as I said in my previous post which you've overlooked, Harman observed the same findings with over-ear headphones (238 listeners) concerning the improvement in reliability and discrimination of trained listeners compared to untrained. And as @pozz says, this all builds on Tools and Olive's similar findings of trained and untrained listeners with speakers. This is well-estsblished stuff by now.
 

ADU

Major Contributor
Joined
Jul 29, 2021
Messages
1,587
Likes
1,086
Well your opinion on IEMs is contrary to Harman's research

Alot of my opinions are contrary to Harman's research. So no big surprises there. :)

Anyway, as I said in my previous post which you've overlooked,

I didn't overlook it. I just didn't respond to it, because there was no context given for it beyond that it involved more participants and over-ear headphones. If or when I can find the time to dig up the study to read more about it, then maybe I'll have some more thoughts on it.

Without a proper context for what I'm looking at though, my thoughts would be mostly meaningless, as in the first example you posted.
 
Last edited:

GaryH

Major Contributor
Joined
May 12, 2021
Messages
1,348
Likes
1,804
I like the idea of learning to listen and training.

I think hearing capabilities play a role as does our ability to accurately specify what we are hearing.

Yet another factor is be able to separate our personal preferences/biases from what's objective. How good are we at self-assessment?

Even after all of that, it is still difficult to communicate one's experience into words. In some ways, objective data is easier because you can just present it, and if you understand what you see, then good. But that's almost never the case. Some people will need help understanding what the data means and how it will translate to a real-world experience. That's where some subjectivity can also come into play when one has to pick and choose which data to focus on and which information carries more weight than what.

In the end, we are humans, not robots. I wouldn't "expect" someone to take my subjective opinions as a "useful data point". I would just think they would see it as a subjective opinion the way I see your suggestion as just a suggestion and nothing more. The subjective weight they would give to my opinion would be based on their trust in me, which is also subjective.

The only way that could be measured objectively might be to see how accurately I've assessed products in the past, but even that requires some translation. I would give myself an 80% accuracy score maybe. B grade. I watched an old review of mine today and thought I was too easy on the product, and would've said something different now. I would give myself an A for effort though. ;-) I'm trying my best.

I also want people to know that the expectations viewers have for us is not the expectations we set for ourselves. I started my channel just to "Show & Tell" stuff I thought was cool. I didn't call myself "New Consumer Reports" or anything like that. I don't know that it's fair to expect all reviewers to be like Audio Science Review. That's why I'm happy this place is here, and Amir is doing what he does. I understand there's some responsibility that goes along with the growth of my channel, but that doesn't mean I should have to change the entire purpose of my channel because of relative success.

So...is that a yes or no to giving Harman's How to Listen program a try? ;)
 

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
924
Likes
1,512
Well your opinion on IEMs is contrary to Harman's research, which found a 0.91 correlation between actual and predicted ratings given by listeners in blind tests, compared to 0.86 for both over-ear headphones and speakers.

You know what GaryH ? Perhaps you should read the actual articles, that way you'd understand how these numbers were derived. They don't quite mean what you think they mean.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
So...is that a yes or no to giving Harman's How to Listen program a try? ;)

How many hours did you invest in this listening program ? Just to get a feel of how much time is involved.
This could be decisive for people to commit (valuable) time if you only accept the highest levels in all tests.
I assume you have otherwise you can not know the actual benefits of this educational program.

Before a few days back I have never been asked about my electronics education level by anyone nor about which 'audio related' education I had and which diplomas, which books, papers I read and which ones not and whether or not I fully understood them.
Also no questions as to how long I have been in (audio)electronics and what my experiences are in the area of music, recording and reproduction.

If I had known doing this program would validate my subjective personal opinions and findings would instantly take away all doubt for those interested in audio I might even have gone through the trouble.
You seem to be the only one asking to do this (in a demanding way) so you can take what I, and now also industry insiders and reviewers as well, write more serious.
 

joentell

Active Member
Reviewer
Joined
Feb 4, 2020
Messages
237
Likes
766
Location
Los Angeles
Your comments probably underlie a lot of what reviewers are thinking. So I'll answer in general.

Look at Amir: the panthers, which started as a funny, lighthearted thing, are now called "panther ratings" by the forum members. People give them numerical value and analyze them. Reviewers, particularly successful reviewers, aren't the same brand of human as everyone else. Their opinions carry a very different sort of weight.

I don't think you have to adopt a rigorous, measurement based show and review process. Your sincerity is not really in question, but your judgments are. And the reliability of your advice and commentary. So to be consistent with your approach, I think your main responsibility is to acknowledge your level of influence. You might be a regular dude but what you say really counts. Help people understand what your opinions are and aren't.
When you mentioned that "people" give numerical values to the panther ratings, that's the viewer and includes you. My point is that this is the Internet, not TV. This is a two-way conversation. Viewers are influencers too. We all switch back and forth between being viewers and content creators ourselves. Each post here is a piece of content. You wouldn't have responded to me if you didn't think there was a chance that you would influence my decision. I just said that my reviews are maybe a B grade. I always tell people to watch other reviews before making a decision. I'm no gatekeeper. And yes, just a dude with a camera and a mic. I'm a hobbyist.
 

joentell

Active Member
Reviewer
Joined
Feb 4, 2020
Messages
237
Likes
766
Location
Los Angeles
So...is that a yes or no to giving Harman's How to Listen program a try? ;)
I've actually downloaded and used that program years ago after seeing it in the Sean Olive slide show presentation. I also have this video queued up. I'm a fan of Dr. Floyd Toole and read his book almost every night. I even got a chance to talk to him on the phone for about an hour a few weeks ago. You might be preaching to the choir with me.

I've mentioned this in a video, but because I do FR measurements of the products I review, I do challenge myself to correlate what I hear subjectively with what the measurements show objectively. I sometimes do something as silly as drawing a FR graph by ear prior to measurement using music or pink noise, then I see how close I can get to the actual measurement after.

I also spend many hours, sometimes 12 hours straight, doing measurements and calibrating my system with PEQ in an attempt to find better ways to do room correction. I get to see the realtime effects of Frequency, Gain, and Q, which also helps train my hearing.
 

Attachments

  • Screenshot_20211028-083049.jpg
    Screenshot_20211028-083049.jpg
    49.2 KB · Views: 73

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
329
Likes
3,004
This is probably a question better posed to Dr. Olive. But I suppose that could be one possible conclusion you could draw from some of their early tests.
Trained listeners have been shown to be more discriminating and consistent in rating loudspeakers and headphones. The How to Listen is useful in training them to identify
Thank you for clarifying this, GaryH.

If you had mentioned this graph was for IEMs to begin with though, then I probably wouldn't have bothered to respond on it. Because imho IEMs are much a less reliable tool for analyzing and predicting users' frequency response preferences than over-ear headphones (or speakers). So I'm not sure why it would be that relevant in this discussion. (?)

Dr. Olive briefly covered this particular study in his presentation for the HBK online conference though. And I've cued the video below to that point, in case others would like to hear his comments and a few more details on it.

Based on his remarks, there were 71 listeners in the study altogether, approximately half of which were trained (as GaryH mentions above). And all of the participants in the study were US Harman employees, with a median age of 35. It appears that the untrained listeners in this study were NOT tested for normal hearing though, like the trained listeners were. Which is interesting, and would tend to introduce more variables into the results imo than if they had also been screened for this.

This was Dr. Olive's last thought on the above test though...



It is true we did not test the untrained listeners for normal hearing as we didn't have the time or budget to do so, and the effect of hearing loss on preference was not the focus of the study. Apart from the usual overall elevated ratings of untrained listeners, the similarity in preference ratings between trained and untrained suggests that if the untrained group had more hearing loss (we don't know this) it didn't have a significant impact on their preferences.

I am collaborating on a research project where the focus is headphone performance above 8 kHz. Hearing loss is considered to be a significant nuisance variable in the study, so all subjects are being screened using HF audiometry up to 20 kHz.
 
Last edited:

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
When you mentioned that "people" give numerical values to the panther ratings, that's the viewer and includes you. My point is that this is the Internet, not TV. This is a two-way conversation. Viewers are influencers too. We all switch back and forth between being viewers and content creators ourselves. Each post here is a piece of content. You wouldn't have responded to me if you didn't think there was a chance that you would influence my decision. I just said that my reviews are maybe a B grade. I always tell people to watch other reviews before making a decision. I'm no gatekeeper. And yes, just a dude with a camera and a mic. I'm a hobbyist.
Despite the lines being blurred these days I'm surprised you don't consider yourself a professional reviewer. Say your position's somewhere in between. Maybe there's not much more to the reviewer's ethics than what you're already doing. Hopefully others will follow suit in that case.

As for me, I don't have any other social media presence apart from Discogs. I try to read a lot and take a while to think about what I've written before posting. Make the words count and not get, you know, too hotheaded during debates. (Forum member's credo right there.)
 

GaryH

Major Contributor
Joined
May 12, 2021
Messages
1,348
Likes
1,804
I also have this video queued up.
I, and I suspect most members here, are more interested in the quantifiable, verifiable science of, rather than the art of listening ;)
I've actually downloaded and used that program years ago after seeing it in the Sean Olive slide show presentation.
Great, and on that quantifiable note, what was the highest level you could pass?
 

ADU

Major Contributor
Joined
Jul 29, 2021
Messages
1,587
Likes
1,086
Trained listeners have been shown to be more discriminating and consistent in rating loudspeakers and headphones. The How to Listen is useful in training them to identify

It is true we did not test the untrained listeners for normal hearing as we didn't have the time or budget to do so, and the effect of hearing loss on preference was not the focus of the study. Apart from the usual overall elevated ratings of untrained listeners, the similarity in preference ratings between trained and untrained suggests that if the untrained group had more hearing loss (we don't know this) it didn't have a significant impact on their preferences.

Thank you for better clarifying some of this, Dr. Olive.

The first part of your remarks seems to have been cut off mid-sentence though. So not quite sure what you were sayin there.

If you believe that the listeners who train with Harman's How To Listen system are more discriminating, then I'd be curious to hear some more about how that conclusion was arrived at. Because it appears to me that at least some of the data gathered from these tests could potentially be interpreted in some different ways.

By "discriminating", I assume you mean that they're better at separating or distinguishing the speakers or headphones in a study based on their different timbral or FR characteristics, resonances and so forth than an untrained listener would be... Which makes perfect sense.

I am collaborating on a research project where the focus is headphone performance above 8 kHz. Hearing loss is considered to be a significant nuisance variable in the study, so all subjects are being screened using HF audiometry up to 20 kHz.

Sounds interesting.
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,554
Likes
1,701
Location
California
If you believe that the listeners who train with Harman's How To Listen system are more discriminating, then I'd be curious to hear some more about how that conclusion was arrived at. Because it appears to me that at least some of the data gathered from these tests could potentially be interpreted in some different ways.

By "discriminating", I assume you mean that they're better at separating or distinguishing the speakers or headphones in a study based on their different timbral or FR characteristics, resonances and so forth than an untrained listener would be... Which makes perfect sense.

Just wondering, have you read the actual papers and do you have the background to be able to interpret them?

…because what you asked is a pretty basic question that suggests no and no
 

ADU

Major Contributor
Joined
Jul 29, 2021
Messages
1,587
Likes
1,086
Well your opinion on IEMs is contrary to Harman's research, which found a 0.91 correlation between actual and predicted ratings given by listeners in blind tests, compared to 0.86 for both over-ear headphones and speakers. Anyway, as I said in my previous post which you've overlooked, Harman observed the same findings with over-ear headphones (238 listeners) concerning the improvement in reliability and discrimination of trained listeners compared to untrained. And as @pozz says, this all builds on Tools and Olive's similar findings of trained and untrained listeners with speakers. This is well-estsblished stuff by now.

Fwiw, here is my attempt at explaining why I think over-ear headphones (and speakers) are generally better than IEMs for these kinds of generic frequency response tests...


These are probably the most salient parts of the above post on this subject...

Ear-resonance.jpg


Since an IEM has the least amount of interaction with the different parts of the anatomy described above, it is likely to require the greatest amount of individualization or customization from person to person to achieve the same sound which is described in that top overall curve, from one individual to the next. And it would therefore also follow that a fullsize over-ear headphone, that completely surrounds the ear, would probably need the least amount of individualization/customization of the headphone types for different individuals to achieve the same sound. Because the sound from an over-ear headphone interacts with more of an individual person's anatomy than the other two headphone types.

The main crux of what I'm saying above though is just that IEMs would likely require the most customization or individualization to produce a similar sound from person to person, because there is less of an interaction between the IEM and an individual's particular HRTF than with the other headphone types (or a sound source in room).

Best to read the entire post though to understand the context of the above statements.
 
Last edited:
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,376
Likes
234,540
Location
Seattle Area
There is a major benefit to compliance with Harman target that is not talked about. That is, if we can get the industry to move in that direction en masse, then we solve the problem of people who create content using a different tonality than those of us who listen. Once there, then people can vary the response to taste if needed but accuracy is accomplished.

No other target curve has a chance of achieving this standardization since it doesn't have the body of research and goodwill that Harman target has.

To that end, whether you agree or not with Harman target, it is a wonderful thing to get behind and achieve standardization.

I think we have started down the path of achieving this target with our consistent approach toward compliance with Harman target. I suspect a few years from now we will have a much larger set of headphones complying with it as to garner our support and praise.

So I say if you care about the larger problem of lack of standardization in audio, you better get behind this movement.
 

ADU

Major Contributor
Joined
Jul 29, 2021
Messages
1,587
Likes
1,086
Just wondering, have you read the actual papers and do you have the background to be able to interpret them?

…because what you asked is a pretty basic question that suggests no and no

I am not an AES member. But have read some of the white papers that are open access. And have also looked at many of Dr. Olive's and also Dr. Toole's presentations on the Harman studies. (And I do mean many.) Which admittedly don't dig down completely into all of the details. Hence my questions, and attempts to better clarify some of the above.

As I explained in this other topic though some time ago, I am self-taught on most of this. And learned most of what I know on the subject of headphones from reading Tyll Hertsen's articles and reviews, and watching his videos (including his presentations for various groups on how to interpret his headphone measurements, and also his articles on the Harman target, some of which disappeared after he retired and Inner Fidelity shut down), and through my own trial and error with various products, and through interactions with other knowledgeable individuals on forums like this one. And also by comparing and contrasting the plethora of frequency response data that's become increasingly available on both headphones and loudspeakers, thanks to websites like this one, and a number of others.

So no, I have no formal technical training in the subjects of either acoustics or engineering, if that's what you're driving at. Or statistics, beyond some basic high school and college science courses.
 
Last edited:

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,878
Likes
6,674
Location
UK
There is a major benefit to compliance with Harman target that is not talked about. That is, if we can get the industry to move in that direction en masse, then we solve the problem of people who create content using a different tonality than those of us who listen. Once there, then people can vary the response to taste if needed but accuracy is accomplished.

No other target curve has a chance of achieving this standardization since it doesn't have the body of research and goodwill that Harman target has.

To that end, whether you agree or not with Harman target, it is a wonderful thing to get behind and achieve standardization.

I think we have started down the path of achieving this target with our consistent approach toward compliance with Harman target. I suspect a few years from now we will have a much larger set of headphones complying with it as to garner our support and praise.

So I say if you care about the larger problem of lack of standardization in audio, you better get behind this movement.
The only downside (or accuracy variation) that is involved with that statement is that not everyone hears Harman Target Headphones with the same tonality, due to people's different HRTF's and also some specific ear to headphone interactions that are not readily predictable.....but I certainly agree that it's a good standard to get behind, even if it's not true complete transparency, but in terms of headphones it is certainly the best target to get behind to help lesson "Circle of Confusion" issues, if we're assuming a lot of music is ultimately created & mastered (if correct terminology) on headphones as opposed to speakers. (But I should add that it also helps to lesson "Circle of Confusion" even if the music has been created using speakers rather than headphones, as we know it's designed largely to emulate Flat Anechoic Speakers in a good listening room).
 
Last edited:

ADU

Major Contributor
Joined
Jul 29, 2021
Messages
1,587
Likes
1,086
I like the idea of learning to listen and training.

I think hearing capabilities play a role as does our ability to accurately specify what we are hearing.

Yet another factor is be able to separate our personal preferences/biases from what's objective. How good are we at self-assessment?

Even after all of that, it is still difficult to communicate one's experience into words. In some ways, objective data is easier because you can just present it, and if you understand what you see, then good. But that's almost never the case. Some people will need help understanding what the data means and how it will translate to a real-world experience. That's where some subjectivity can also come into play when one has to pick and choose which data to focus on and which information carries more weight than what.

In the end, we are humans, not robots. I wouldn't "expect" someone to take my subjective opinions as a "useful data point". I would just think they would see it as a subjective opinion the way I see your suggestion as just a suggestion and nothing more. The subjective weight they would give to my opinion would be based on their trust in me, which is also subjective.

The only way that could be measured objectively might be to see how accurately I've assessed products in the past, but even that requires some translation. I would give myself an 80% accuracy score maybe. B grade. I watched an old review of mine today and thought I was too easy on the product, and would've said something different now. I would give myself an A for effort though. ;-) I'm trying my best.

I also want people to know that the expectations viewers have for us is not the expectations we set for ourselves. I started my channel just to "Show & Tell" stuff I thought was cool. I didn't call myself "New Consumer Reports" or anything like that. I don't know that it's fair to expect all reviewers to be like Audio Science Review. That's why I'm happy this place is here, and Amir is doing what he does. I understand there's some responsibility that goes along with the growth of my channel, but that doesn't mean I should have to change the entire purpose of my channel because of relative success.

Good to see you and some other reviewers on board here, joentell. Always enjoy your vids.
 
Top Bottom