Master Complaint Thread About Headphone Measurements

[IAtaman]

Why though? It's clear you would like to see it, but how would it cater to ASR mission?

As far as I understand it, science (in general) demands adherence to a repeatable methodology and the simpler the result of research, the more applicable / useful it is. Especially in the context of our extremely convoluted industry. Accommodating variations (like you suggest) would make the results more complicated, diluted and less accessible to the public.

I think your prerogative is to make ASR research cater to a wider audience (which is a noble goal in itself!) but what you propose could actually end up perpetuating the current status quo. What do you think?

Thank you for not assuming I have a nefarious agenda hiding under my comments, and engaging contructively to understand my point of view.

I am not sure what I am suggesting is the right thing to do. So far no one seems to be on board with it so there is a good chance it might not be

I am suggesting it because I think it would make the evaluation more relatable to public. "This headphone is not compliant to target, but it is tonally balanced, and does not have any major red flags. You might be really enjoying its tuning, and science have an explanation for it too."

I don't think it would end up perpetuating the current status quo, if you take status quo to mean "more expensive = more better, lets try to find ways to explain why that is".

If you mean it would bring in subjective elements back, I think there might be ways around that as well. Research uses what is effectively "tilt of the error curve" as one of the parameters that predicts preference for example, which might indicate the overall tonal balance of the headphone. If a headphone's error curve is tilted in either direction significantly, that might mean the headphone's tuning is not balanced objectively for example. Or research shows too much energy in mid-bass is not liked across the board - too much mid-bass = objectively bad. Allowing for preference does not mean everything goes.

 

[solderdude]

If that is indeed the case then that means on average untrained people, under controlled test conditions, find HD800S's out of the box tuning as preferable as the target itself while listening to music lacking in sub bass.

Many people love the HD800(S) out of the box.
That does not mean it can't improve even more when you EQ them.
I am on the same boat. I love it (HD800) ... with EQ.. just not the same EQ as Oratory nor Amir recommends though.
I don't mind, people differ, preferences differ.
I don't care if my ears prefer another tonality as Harman found nor what people in the research thought of the emulated (or real) HD800(S) and certainly won't call out Amir on my preference.

The fact that Amir does not like a headphone or like it as much without EQ is another matter. It's his review. It is not a democracy where the majority has to agree it is crap or not.

his original disclaimer was very good and maybe can be reused

Just kindly ask Amir if he wants to include his disclaimer again. When Amir does not want to.. that's it.

 

[markanini]

Including discussion of subjective impressions and judgements is done out of necessity becuse of elements of variability due to fit, limited coupler-to-eardrum matching, to name a few.

I'm divided on preference bands, I like what it could represent. If it was based on concrete data it opens a number of questions if you want to make a serious effort. How much will the bands account for fit and how much for preference? How much effort is worth putting in? Should you sample a number of transducers of varying types, on-ear, around-ear, in closed-back and open-back variants, and in-ear. And calculate separate bands for each type?

Maybe an alternative would be to take new steps to educate about the realities of measurement tech. Amir reviews link to a video "Understanding Headphone Measurements" near the start. It wound't be bad if the bullet points could be condensed and presented more frequently on various platforms, because attention spans vary and everyone is not on ASR or it's youtube channel.

 

[Pe8er]

"You might be really enjoying its tuning, and science have an explanation for it too."

(Emphasis mine) I think this is where you might be going in a dangerous direction. How is the reader supposed to determine whether they will enjoy this tuning or not? If every kind of tuning gets this type of review, then anything might be good until one tries it, which for most people mean purchase it, with varying degrees of opportunity to return. That doesn't make the search any easier and is downright dangerous for the wallet. This is what the industry wants, the status quo I mentioned.

ASR's mission, as I understand it, is to put an as objective as possible frame on an industry where marketing slows down engineering progress and harms customers. To evaluate mechanically engineered objects that conform to laws of physics with tools from the exact same domain and nothing more. To lead a struggle against ambiguity (read it in Slavoj Zizek's voice haha). And try to keep things as simple as possible in a very complicated environment.

Whereas your "might be" takes us a step backwards. It may make more headphones or speakers seem more desirable to a wider audience, but in reality diminishing the frame of reference eliminates the value of the review / recommendation.

 

[OnLyTNT]

I am sorry, I did not mean to ignore your comments but was carried away by other discussions

It is ok, no need to be sorry.

Yes, it does look indeed similar to HE400SE. HE400S was one of the headphones studied, maybe it might be that one, I am not sure. I heard some people say it might be the HD800S as well.

If that is all there is to the reviews then why all the listening tests and the written comments and such, why not share PDF files like Oratory1990 does and let everyone decide for themselves?

Reviews come in different formats and styles. A review of full of numbers and graphs is kind of dull and might be intimidating for average user. Personally and honestly, I mostly do a quick check on the graphs and if a question occurs in my mind or can't get a full grasp about what I see, I check the comment. Some of the comments and small notes can be useful or entertaining. Some personal comments and humour doesn't hurt, no need to be too strict and it doesn't deteriorate the objectivity in my opinion. I don't always check listening tests and only do it sometimes to see if the graph aligns with the listening experience to make sense. Vice versa, my own listening test mostly and "roughly" aligns with the graph. In general I do what I do when I read a paper which is try to get most useful information quickly (if it is not a critical one, of course...). Some people like to read reviews word by word by heart. So, not only writing but also reading reviews comes in different styles.

Of course you can. Do all your benchmarking of graphics cards using Blender Cuda and see how the results look. Or take any Apple presentation on the performance of their hardware vs that of competition. Do you think those are objective? For the record, that is not what Amir is doing for sure. I do fully trust the reliability of his measurements. My point is you can manipulate measurements to look a certain way, even in computer hardware world.

Again personally, I don't watch any manufacturer presentation or trust their data, I don't care how reputable the company is. Sure you can manipulate measurements and when you get busted you throw your reputation out of the window but nowadays reputation also does not play much role any more I guess. In the past there were many scandals in which hardware review sites got involved (may be still there are, I don't follow any). nVidia was playing tricks on the drivers to get better results and they got busted but there they are, still going really strong. That's why, to be on the safe side, one should do cross check with multiple measurements (not only measurements actually, do it as much as you can with every information). Especially today's world we are heavily under manipulation form all sides. So, we need as much as objectivism as we can get. Amir also mentioned that he does cross checks with other results to see if his own measurement has a flaw or not. That's a good practice and trustable because he knows we also can do the check and if there is an error we get our hands around his neck (kidding ).

I think I do. You read the recommendations and if it makes sense to you, you go for it. What about the products you did not go for, because the conclusion was not very positive. Did you make any attempt to listen to them? If you get a chance to listen to them, do you think you'd be able to evaluate whether you like them or not without any bias?

I don't think I can evaluate any headphone by listening them without any bias. Though, I would like to try to see how awkward it will be for me but I can't because most of the reviewed products are not available not only the city I live in but the entire country. At least I can say this: when I try a headphone and if I don't like the tonality I don't hate it with my all guts unless it is not outrageously bad. After some listening I get used to it. Until I apply EQ to make it close to the target, then I realize the out of box tonality is definitely not preferable for me.

 

[IAtaman]

(Emphasis mine) I think this is where you might be going in a dangerous direction. How is the reader supposed to determine whether they will enjoy this tuning or not? If every kind of tuning gets this type of review, then anything might be good until one tries it, which for most people mean purchase it, with varying degrees of opportunity to return. That doesn't make the search any easier and is downright dangerous for the wallet. This is what the industry wants, the status quo I mentioned.

I was considering the point of view of someone who already owns the headphone. From the perspective of a person who does not own it, they can do it in whatever form they are doing to determine whether they will enjoy the tuning. In any case, a conclusion like "not compliant to target, but it is balanced, if you like less bass, you might find it preferable" might be more helpful in my opinion.

ASR's mission, as I understand it, is to put an as objective as possible frame on an industry where marketing slows down engineering progress and harms customers. To evaluate mechanically engineered objects that conform to laws of physics with tools from the exact same domain and nothing more. To lead a struggle against ambiguity (read it in Slavoj Zizek's voice haha). And try to keep things as simple as possible in a very complicated environment.

Whereas your "might be" takes us a step backwards. It may make more headphones or speakers seem more desirable to a wider audience, but in reality diminishing the frame of reference eliminates the value of the review / recommendation.

I am not advocating for ambiguity. Variation in preference is not the same as ambiguity.
Here is a link to an article written by Dr. Olive himself that should make things a bit more clear in my opinion, and here is a direct quote from that article.

Altogether, these findings suggest that a single headphone target may not be sufficient to satisfy variations in the recordings, individual tastes, listening experience, and hearing loss.

I think accommodating for these variations in reviews is bound to make the science and the reviews more relatable to more people. That does not mean everything will be recommended in one way or another.

(I did try to read that it in Žižek's voice, emphasizing "struggle". I am not sure if I was able to do him justice though )

 

[splitting_ears]

I am not advocating for ambiguity. Variation in preference is not the same as ambiguity.
Here is a link to an article written by Dr. Olive himself that should make things a bit more clear in my opinion, and here is a direct quote from that article.

Altogether, these findings suggest that a single headphone target may not be sufficient to satisfy variations in the recordings, individual tastes, listening experience, and hearing loss.

I think accommodating for these variations in reviews is bound to make the science and the reviews more relatable to more people. That does not mean everything will be recommended in one way or another.

It's a sensible concern, but I'm afraid the solution to these real (and legitimate) variations won't come from different review angles. Far more helpful will be the widespread implementation of accessible EQs or simple bass/treble tone controls for the end user.

From the same paper : "The two smaller classes of listeners who prefer headphones with more bass or less bass can be accommodated through a simple bass tone control on the headphone or via an app on the audio device."

 

[IAtaman]

It's a sensible concern, but I'm afraid the solution to these real (and legitimate) variations won't come from different review angles. Far more helpful will be the widespread implementation of accessible EQs or simple bass/treble tone controls for the end user.

From the same paper : "The two smaller classes of listeners who prefer headphones with more bass or less bass can be accommodated through a simple bass tone control on the headphone or via an app on the audio device."

Yes, I agree. EQ is the solution to most headphone problems in life Products like Free DSP Cables can hopefully make it a lot easier to EQ to taste for everyone no matter what source they use. 50% of 50+ year olds prefer less bass, I am sure you have seen the table in the article. One way of accommodating the variation in preference could be offering EQ settings with 3dB less bass with a still well balanced tuning for example, wouldn't that review be more interesting and have more followers from 50+ demographic?

 

[Thomas Savage]

I'd not normally encourage a @amirm analogy but sometimes you burn the rice and sometimes the rice .., kills you ... so let's burn the rice and work backwards ...

 

[OnLyTNT]

By the way, there is something I would like to add about the target. Just some thoughts.

You know there are some atomic models or theories, and most accepted one is "the modern atomic theory". They are all "theories" (check their chronological order) and widely accepted one is the latest "modern" one. You can accept it or not, that is the best we have and if you have any better model then there is probably a Nobel waiting for you.

Harman research is the best we have as scientific as it can go. Also, It is a "statistical model" and of course there will be deviations from the model because it "predicts" your "preference". "A Statistical Model that Predicts Listeners' Preference Ratings of...", well, it says in the title . The model's prediction may fail for some listeners but statistically it is significant and good enough to be chosen as a target. Lastly, when it comes to "data analysis" you may apply some restrictions to your research. It may affect your results or not and the impact can be calculated so you can check if your analysis and the result are good enough or acceptable with or without those restrictions.

As a conclusion, until a better or more "comprehensive" model comes up (I doubt), our beloved target is what we have.

 

[amirm]

If that is all there is to the reviews then why all the listening tests and the written comments and such, why not share PDF files like Oratory1990 does and let everyone decide for themselves?

So you would be in favor of asking your doctor to only hand out diagnostic reports, imaging, etc. and let patients decide for themselves? You don't want him to be an experienced, trusted agent to opine on said reports and tell you what could be wrong with you? Just because such opinion is not based on 100% certainty?

Or maybe when you take a mechanic to evaluate a used car, you want him to just give you a measurement report. That he should refrain from telling you if you should or should not buy the car.

So no, that is not what we are going to do. From day one I have annotated my measurements to point out significant areas, good and bad. This allows people to learn what the graphs/measurements are saying. None of those notations take out the underlying data. Readers can still choose to interpret them differently. And many are especially in testing of headphones and speakers.

The whole notion of this website was to bring science down to a level that could be understood. That comes from interpreting results and creating an environment where members can keep getting more and more educated.

And it is not like you have come here with some data/research that your suggestion is sound. You have an agenda to make more opinions right about a headphone. So many manufacturers would want that. And is what many other reviewers do with statements like, "it doesn't measure right, and it didn't sound great to me but hey, your opinion and preferences is different so you may like it." What the heck did I read that review for if at the end I was to decide by myself? And that anything goes?

So no, I am not going to take advice from you. The system we have is working exceptionally well. I measure, perform subjective evaluation and EQ, and conclude with some personal remarks. Then an entire thread is dedicated to people voicing differing opinions. Or approval. Members have full access to all of these opinions which again, is different than what many reviewers do. It is by design. Every review invites criticism, comments, questions at the end of the review. You want this changed because you think so? Well, I don't think so.

 

[amirm]

What about the products you did not go for, because the conclusion was not very positive. Did you make any attempt to listen to them? If you get a chance to listen to them, do you think you'd be able to evaluate whether you like them or not without any bias?

Answering the last part, most likely your evaluation would be incorrect. Simply listening to a headphone doesn't tell you much about its fidelity. This was discovered way back by Dr. Toole. That since we do not have a reference, you can't properly judge the sound of a transducer. The problem was solved in multi-way testing. Put 4 speakers in the test and your ears instantly get calibrated by the range of tonalities you are hearing. Then, a speaker that is brighter than the rest stands out like a sore thumb. By itself, how do you know it is not your music that is bright?

In many instances the immediate impression of what I think of a headphone is proven incomplete once I develop EQ and do AB tests. It is only then that I realize how well the stock tuning sounds. And unlike the above evaluation, I can perform this AB test blind.

The danger in what you suggest is that a faulty test may talk someone into keeping a multi-thousand dollar headphone. Then one day they compare it to a more performant one and realize it was a mistake.

The job of my reviews is to narrow the field down to those products that perform well and in many cases priced very competitively. You are still welcome and even encouraged to listen for yourself and have return privileges if you don't like it. But now we are working as a team, with me, the science of headphone preference and measurements all working together to give you a high confidence answer.

Can we still go wrong? Yes. Once again, since we lack standards in audio, there is some ambiguity here that can never be ironed out. But it is a heck of a lot better than relying on completely error prone process.

All this said, we are not running a police state here. Do whatever you like to do. But don't tell me and the rest of us that personal listening somehow leads to a superior selection process. We exist because that is not true.

 

[IAtaman]

So you would be in favor of asking your doctor to only hand out diagnostic reports, imaging, etc. and let patients decide for themselves? You don't want him to be an experienced, trusted agent to opine on said reports and tell you what could be wrong with you? Just because such opinion is not based on 100% certainty?

Or maybe when you take a mechanic to evaluate a used car, you want him to just give you a measurement report. That he should refrain from telling you if you should or should not buy the car.

So no, that is not what we are going to do. From day one I have annotated my measurements to point out significant areas, good and bad. This allows people to learn what the graphs/measurements are saying. None of those notations take out the underlying data. Readers can still choose to interpret them differently. And many are especially in testing of headphones and speakers.

You are taking things out of context and putting words into my mouth. I did not suggest the commentary should be removed, I was responding to the remark made that people can make up their own mind from objective measurements. I imagine it would be clear to anyone who is reading to genuinely understand that after suggesting you bring back your "you don't have an exact idea based on graphs alone" disclaimer, I am not suggesting to remove the commentary.

The whole notion of this website was to bring science down to a level that could be understood. That comes from interpreting results and creating an environment where members can keep getting more and more educated.

And it is not like you have come here with some data/research that your suggestion is sound. You have an agenda to make more opinions right about a headphone. So many manufacturers would want that. And is what many other reviewers do with statements like, "it doesn't measure right, and it didn't sound great to me but hey, your opinion and preferences is different so you may like it." What the heck did I read that review for if at the end I was to decide by myself? And that anything goes?

So no, I am not going to take advice from you. The system we have is working exceptionally well. I measure, perform subjective evaluation and EQ, and conclude with some personal remarks. Then an entire thread is dedicated to people voicing differing opinions. Or approval. Members have full access to all of these opinions which again, is different than what many reviewers do. It is by design. Every review invites criticism, comments, questions at the end of the review. You want this changed because you think so? Well, I don't think so.

Why are you trying to make enemies out of everyone I don't understand.

I do not have a secret agenda. I am just a consumer, an enthusiast and a member of this forum for almost 3 years now. I explained my agenda, I would like more people to see Harman research as the reference for what sounds good. I am not advising anyone anything. I am sharing my opinions on how the headphone science can be made more attractive to more people through more inclusive reviews. If you don't agree, you can respond, or you can ignore. But there is really no need to make unfounded accusations about my intentions or present that I am saying something I am not by selectively quoting my remarks out of context.

 

[splitting_ears]

Why are you trying to make enemies out of everyone I don't understand.

I understand that Amir's posts may sound harsh, but I'd suggest that you put this particular issue to rest for a while and see if your opinion changes/evolves over time. You've made yourself perfectly clear, and that's not lost for sure

About two months ago I started another (unrelated) thread advocating intersample peak testing in DACs. I really wanted people to like the idea, and I really thought it could be worthwhile for ASR as a whole. A few members were helpful and interested, many others weren't, and some were quite defensive/dismissive. The latter was a bit difficult for me as I really dislike conflict. Later in the thread, Amir replied to some of my questions (even though I was virtually nobody here) which surprised me. I've learnt a lot by sharing and expressing my opinions, and then the topic lost traction. I could have kept posting to try and get more people involved, but I didn't because I didn't want it to become my own, personal battle.

The good news is that I'm still happy to browse the forum and participate from time to time. I've also read other reviews and watched some of Amir's videos where I've been able to better understand his way of thinking in the meantime - in my case, the benefits of keeping DAC reviews short and not bombarding the reader with graphs everywhere.

If other people wonder about ISP testing in the future, my thread and many others are still there for reference and they can pick up where I left off. It can only work if other people bring their own arguments too.

I guess what I'm trying to say is that there's no rush to fix every flaw we see, especially if it means interacting in an unfriendly way with people we hold in high esteem. There's always something more to learn that might change our opinions for the better later on.

 

[Chagall]

I understand that Amir's posts may sound harsh, but I'd suggest that you put this particular issue to rest for a while and see if your opinion changes/evolves over time. You've made yourself perfectly clear, and that's not lost for sure

About two months ago I started another (unrelated) thread advocating intersample peak testing in DACs. I really wanted people to like the idea, and I really thought it could be worthwhile for ASR as a whole. A few members were helpful and interested, many others weren't, and some were quite defensive/dismissive. The latter was a bit difficult for me as I really dislike conflict. Later in the thread, Amir replied to some of my questions (even though I was virtually nobody here) which surprised me. I've learnt a lot by sharing and expressing my opinions, and then the topic lost traction. I could have kept posting to try and get more people involved, but I didn't because I didn't want it to become my own, personal battle.

The good news is that I'm still happy to browse the forum and participate from time to time. I've also read other reviews and watched some of Amir's videos where I've been able to better understand his way of thinking in the meantime - in my case, the benefits of keeping DAC reviews short and not bombarding the reader with graphs everywhere.

If other people wonder about ISP testing in the future, my thread and many others are still there for reference and they can pick up where I left off. It can only work if other people bring their own arguments too.

I guess what I'm trying to say is that there's no rush to fix every flaw we see, especially if it means interacting in an unfriendly way with people we hold in high esteem. There's always something more to learn that might change our opinions for the better later on.

That's a healthy way of dealing with this kind of disagreement for sure.

I just hope people (in however high esteem) will be a bit more understanding in general. The irony is threads like "Let's be a little nicer, especially to newcomers" exist, but in everyday discussions, people pay it no mind and return to their default reactions of attack and dismissal more often than not. A shame really.

A big thank you to all the exceptions to this rule!

 

[stalepie2]

Pro headphones meant for mixing and mastering shouldn't be judged as music-listening headphones. They shouldn't have a lot of extra bass like the Harman curve and consumer phones. They're not supposed to be enjoyable, but a tool used to mix music, which is why they're called "mixing" headphones. It's why when you're on the company's website, like at Beyerdynamic's site, and you're looking at the product pages for headphones like DT 900 and DT 880 Pro, it says they're good for mixing and mastering audio. They have extra space between the sounds (too much separation) and a flat sound (which is boring, not much fun) and then you check levels for different tracks to make sure they're mixed together well. I just think these kind of headphones shouldn't be graded the same way as consumer music-enjoyment headphones sold at like Best Buy and walmart. I see a lot of headphones judged that way on the reviews at this site and it makes no sense.

 

[amirm]

Pro headphones meant for mixing and mastering shouldn't be judged as music-listening headphones.

Those folks should do their own studies demonstrating which headphone allows them to make better mix decisions.

Until then, listeners prefer the extra bass because that is what "real" sounds to them. By mixing against that consumer target, then we get standardization of music creation and playback. Using an entirely different response for mixing/mastering music means that we don't hear what they heard. This, by definition is wrong.

 

[stalepie2]

Those folks should do their own studies demonstrating which headphone allows them to make better mix decisions.

Until then, listeners prefer the extra bass because that is what "real" sounds to them. By mixing against that consumer target, then we get standardization of music creation and playback. Using an entirely different response for mixing/mastering music means that we don't hear what they heard. This, by definition is wrong.

I can't imagine how a study would be constructed to prove if better mixing decisions are made. That's not exactly something you can put a scientific judgment to. Can you imagine how such a study would be done? Who would be the judges of whether one mix is better than another? And this becomes scientific fact?

The studio monitors have always tried to be flatter than the consumer equipment which is more fun to listen to. If anything, the more fun the headphone is to listen to, the more the music engineer would get carried away in enjoying the music instead of working hard to make sure the recording is mixed well. I don't mean that as a joke. He'd sit there bobbing his head along to the rhythm instead of keeping his mind on his work. The studio headphones need to sound flat and dead and be more "analytical." They're supposed to tear through the sound to find problems in the mix. Like if the treble is too peaky or other boring stuff like that.

 

[splitting_ears]

He'd sit there bobbing his head along to the rhythm instead of keeping his mind on his work.

Actually, this happens to me quite often when I work on other people's music

 

[oleg87]

For lack of a better place to ask... is there some page here that documents Amir's test protocols, for example, what frequency is used for headphone sensitivity and amp power measurements?