• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Poll: Should We Get Into Testing Headphones or Speakers?

Should ASR get into testing speakers, headphones, or neither for now.

  • Speakers

    Votes: 145 56.0%
  • Headphones

    Votes: 77 29.7%
  • Neither. Can look again in a year or two.

    Votes: 35 13.5%
  • Never

    Votes: 2 0.8%

  • Total voters
    259

oivavoi

Major Contributor
Forum Donor
Joined
Jan 12, 2017
Messages
1,721
Likes
1,934
Location
Oslo, Norway
«The Harman curve», you wrote. Which of them? There have been more than one Harman curve. My forecast is there will be more.

You’re a researcher and scientist. How do you normally respond to references that are drifting? And this is not social sciences where drift is inherent. This is primarily the «harder» sciences. Humans don’t evolve from year to year, as the Harman curve does.

Yeah, good question. When it comes to my own listening and purchasing decisions, at least, I have a very pragmatic approach to this. So I kind of assume that I'm not that unique, and that there's a good chance that I'll like the same kind of curve that the average listener in Harman's listening panels like. Then again, I have heard some speaker systems which were equalized to the Harman house curve, and I have often perceived them as too bass-heavy for my taste, so who knows...

As to whether the Harman curve should serve as the benchmark for evaluating headphones, I don't have a strong opinion either way. This is one of the subjects I haven't bothered to learn much about, so my opinions don't matter much.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
The funny thing about 'extra bass' is that while it sounds impressive at first listen and even for quite a few days (new toy syndrome) after a certain time it can become fatiguing and a bit less is desirable.
Usually when something is 'new' one tends to revisit familiar music and plays that louder to hear 'more' as well.

Indeed humans don't evolve but insight in perception does.
 

svart-hvitt

Major Contributor
Joined
Aug 31, 2017
Messages
2,375
Likes
1,253
Yeah, good question. When it comes to my own listening and purchasing decisions, at least, I have a very pragmatic approach to this. So I kind of assume that I'm not that unique, and that there's a good chance that I'll like the same kind of curve that the average listener in Harman's listening panels like. Then again, I have heard some speaker systems which were equalized to the Harman house curve, and I have often perceived them as too bass-heavy for my taste, so who knows...

As to whether the Harman curve should serve as the benchmark for evaluating headphones, I don't have a strong opinion either way. This is one of the subjects I haven't bothered to learn much about, so my opinions don't matter much.

My point is as much - if not more - on methodology.

Any scientist should know about the measurement without theory controversy 75 years ago, just as WW2 was about to end. It was old Mitchell vs. young Koopmans. Koopmans won, which I think was an unfortunate thing, in this particular case. Look it up. Interesting story.

Goethe was on the same path much earlier:

«Content without method leads to phanaticism.
Method without content to empty sophistry», he wrote.

So this discussion is also about looking for potential red flags if science is your aim.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,597
Location
Central Fl
Another vote for measurement of AV-Multich gear. ;)
 

DDF

Addicted to Fun and Learning
Joined
Dec 31, 2018
Messages
617
Likes
1,355
@svart-hvitt , heres the thing about the Harman headphone curve, which inherits the same uncertainty that plagues in room loudspeaker targets. We dont know how to objectively eq a signal for different room contributions (the combined effect of on and off axis radiation, and room contribution) to arrive at a true neutral response. Not even for a given room and DI, never mind different rooms and DIs. In fact, we know its not even quite possible.

So Harman wisely put some stakes in the ground: first try to define what is a neutral speaker in a room. There is no way to DBT this because you dont have access to the original performance space and because by necessity the best (neutral) recordings are artistic attempts, not purely objective, interpretations. So, using DBT, they arrived at targets for speaker design that have consistently shown best outcomes for trained listeners attempting to allocate highest score towards perceived neutrality (one has to appreciate the lack of anchor and the limitations of auditory memory), testing various responses and DIs. They did their best to study if the outcome varied by gender, age, experience, culture, and rooms. The outcome held across these variables.

Based on this, they had a pretty solid theory on what statistically a neutral speaker measured like.

They then took the best available representation of this speaker and placed it in a room meant to be representative of an average listening room. I believe its an IEC room. I've commissioned an IEC room, had it built, and tested its acoustics, so I'm satisfied for myself that its a reasonable choice. They then treated it so it didnt have any gross acoustic anomolies.

As I understand it, they then used a dummy head to record the in room response of this speaker in a most neutral placement. Different dummy heads arrive at different HRTF contributions and its true hrtf varies significantly by user but the dummy heads objectively attempt to represent a best average. They then ran DBT with trained listeners using similar protocols that were used to arrive at the target speaker response. The limitarions of these tests are the same as for the speaker tests: lack of a true anchor for which to A/B against and the limitations of auditory memory. However in acknowledgement that we don't quite know how room response affects perceived tonal balance, they allowed users to adjust tone controls towards achieving perceived neutrality. They then tested the veracity of their results against external variances in the test subjects, and the outcomes held up well.

None of this is perfect, and the limitations bear repeating: no solid A/B anchors, limitations of auditory memory, assumed IEC room, specific recordings chosen, averaged hrtf. But they tested variance across these variables much more deeply than most studies, and the results held up robustly. Not perfectly, but well. That's an important point to keep in mind. So, eq is always a welcome, necessary tool on the playback end.

You are quite right that individualized HRTF creates the need for eq, but it looks like you're missing the point that the Harman curve gives the best average baseline to eq from, and the best shot at neutral sound for the general population if eq cant be applied.

Individualized Hrtf are great but even if you can test it correctly, you're left with a stark choice:
1. Free field calibration, where the hrtf characterization ignores room contribution. We know thats not quite right
2. Diffuse field calibration that integrates in all the room sound. We also know thats not quite right because a long term average curve ignores how the hearing system uses time windowing to build up a representation of timbre. It will also cause the measurements to inherit the non ideal aspects of your room's acoustics. As Dr Olives headphone study showed, some eq is necessary to tweak the room contribution

I think the Harman outcomes are a major gift to those that seek the most neutral reproduced sound possible. But they do have the limitations discussed. Hopefully future testing will refine these conceptts, but can you propose a better way forward?
 
Last edited:

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,597
Location
Central Fl
My nicest setup is with a middling AV receiver and I’m very happy with it.
My system would be considered udder garbage by many of our members here and in a direct comparison it is.
But I'm not sure what any of that has to do with the issue being discussed? Bottom line is Amir has a pretty good handle on what aspects are important to measure in multich AV and has shown that in the prior bench-tests he's done in the past. My sole criticism had been not using a HDMI input for the measurement since that is what the majority of it's owners will use..
I think an AV receiver would have been most relevant to the average consumer a few years ago, but now we are moving on to self-powered wireless technologies (Sonos, etc.) as the mainstream home technologies.
So once again I'm missing your point. If these devices are now the tech of interest, then all the work that has been done prior here was a waste of time. No one is interested in separate DAC's, Amps, Headphone amps, etc, and the whole website has to refocus itself? To the fact that lifestyle gear has become very popular with a certain group, these are not people that, in the main, have any interest in SOTA gear or hang out on tech forums like this.

Moving on I would just like to point out to Amir that there are quite a few of the members here that have systems heavily based on multich reproduction and I believe that percentage can be seen across the entire high end market of today. The Marantz 8805 review sent some waves out on the other multich sites that brought interest to ASR. There isn't really a ton of higher end AV gear when looking to the more serious side of the hobbie. Forgetting the ton of cheap AVR recievers out there, the pre/pro's are where the more SQ dedicated folks are. They allow the use of either separate DAC's and amps, or todays powered speakers Laying hands on the units from Denon, Marantz, Emotiva, NAD, Anthem, Classe, Integra, etc and comparing where they stand in the most basic of chores like a clean stereo playback, the performance of its multich DAC, THD+N. SINAD, etc would be a wonderful thing to know. Evaluation of the performance of things like Dolby, DTS, upsampling of stereo, etc are almost totally in the subjective realm and best left for others to try an evaluate.
Just some thoughts to chew on Amir and a very small number of components compared to other things being discussed.
Cheers
 

Chrise36

Major Contributor
Joined
Dec 23, 2018
Messages
1,043
Likes
359
Hi Amir i think you should be testing speakers and amplifiers.Dacs have good enough specs in my opinion and speakers and amplifiers effect the sound reproduction more.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,597
Location
Central Fl
Can't measuring only things that we know will have no audible effect come to resemble the drunkard's search or the streetlight effect--looking where the light is best when what we are looking for (here, sound quality) is somewhere else?
You are correct from that point of view. But there are many sites that explore that side of things, ignoring what value there might be in their sighted results. Here I belive a large part of what Amir has attempted to present is the technological SOTA in various technologies, leaving it to others to attempt to put some SQ relationship to the numbers. Without extremely time consuming DBT's' involving more listeners than just himself, it would be impossible to make "accurate scientific" proclomations as to the SQ of any particular component.
As far as a stereo system one may own being "udder garbage," at least maybe you can sell the milk. Yes. You did miss my point, and in large part I was agreeing with you, but based on your response perhaps I am best off not engaging with you anymore.
We seem to have a failure to communicate.
1: Yes, then I'm still having a problem trying to understand the points you were trying to make. ?
2: Sorry, in no way was my response meant to be aggressive or combative. I apologize if I offended you in some way. My writing style is short and to the point and sometimes lacks the smoothness others possess.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,597
Location
Central Fl
If we are going to measure stuff way past audibility and lust after numbers and pay more for the best measuring equipment, regardless of its relevance to audible consequences, that's not so different from what the "audiophiles" we ridicule so much do.
But remember what's been revealed here a bunch of times is that SOTA in measurements can be obtained at a very low cost. DAC's costing $100 have beaten ones costing 20 times more. Amps like the Benchmark at $3k, though not peanuts, is still a very low priced product against what the "high end" builders are charging. In Stereophiles Recommended Components the AHB2 is given the $$$ "bang for the buck" award .It's the lowest priced Class A amp in the ratings. Compare that to Mikey Fremers rave baby, the $170,000 darTZeel monstrosity. o_O
So while measuring for SOTA numbers, everyone has learned the best can be had for surprising little.
 

StevenEleven

Addicted to Fun and Learning
Forum Donor
Joined
Dec 1, 2018
Messages
581
Likes
1,188
We seem to have a failure to communicate.
1: Yes, then I'm still having a problem trying to understand the points you were trying to make. ?
;)
 
Last edited:

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
Not Amir, but can say it is an easier task to set up and requires less fiddling.
The reason is that insertion depth, seal, fit and ear canal shape aren't the same for each individual but can be very similar/repeatable for test fixtures.

Making/buying a test fixture is cheaper though and results are easier to publish.
The raw measurements must be compensated and have little meaning when there isn't a good correlation with how the sound is perceived due to the mentioned differences.
 

Nango

Major Contributor
Joined
Aug 6, 2018
Messages
1,463
Likes
985
Location
D:\EU\GER\Rheinhessen
But one could say whether one IEM is more transparent than others despite the individual sealing, etc.?? They sell today more IEMs than headphones with only two competitors in the world manufacturing the drivers (Knowles, Optoma) ..... No one knows if the driver are identical in 50$ IEM and 1.000$ IEMs.
 

fulffy512

Member
Joined
Jun 1, 2019
Messages
46
Likes
21
I think if doing headphone/IEM measuring doing 2 tests with 1 being 90 - 100db and another under 80db.
 

Mad_Economist

Addicted to Fun and Learning
Audio Company
Joined
Nov 29, 2017
Messages
543
Likes
1,618
2. Diffuse field calibration that integrates in all the room sound. We also know thats not quite right because a long term average curve ignores how the hearing system uses time windowing to build up a representation of timbre. It will also cause the measurements to inherit the non ideal aspects of your room's acoustics. As Dr Olives headphone study showed, some eq is necessary to tweak the room contribution

This is an interesting criticism of DF, which I don't believe I've seen before, but it seems a bit out of alignment with ex. Theile's arguments for the diffuse field target. Could you point me in the direction of some citations/research on this? In principle, it doesn't seem to me that time windowing should necessarily prove problematic here, as it was the lack of directional HRTF cues was the purpose of diffuse field (since, if you follow Theile's line, they inevitably lead to timbral errors), but I may be missing something.
 

frogmeat69

Addicted to Fun and Learning
Forum Donor
Joined
Apr 25, 2018
Messages
951
Likes
1,710
Location
Western New York, USA
Doesn't Solderdude measure headphones? Maybe Amir and he could help each other for the good of their respective communities?
 

Frank Dernie

Master Contributor
Forum Donor
Joined
Mar 24, 2016
Messages
6,445
Likes
15,778
Location
Oxfordshire
I must say, I am really surprised by the results of the poll so far, with nearly 70% asking for speaker testing. I thought for sure our audience would be far more biased toward headphones at this point.
My guess is that the probably younger and less experienced people are headphone listeners and the old fogeys who have been playing around with hifi for decades mainly listen to speakers.
The young ones are much more active on forums and have far more questions due, perhaps, to their lack of experience. This leads to far more posts about headphones and headphone amps than speakers and power amps giving a false impression about the overall interests of the forum membership.
I have had headphones for almost 50 years now but never listen to them by preference, only when speaker listening is impossible or intrusive, which for me nowadays is rarely.
I listen to speakers several hours per day.
I never could listen to music when working, either I was totally distracted by the music and giving my business poor value for my time or I was concentrating so much on what I was doing I didn't remember hearing mosty of the music, either way, pointless for me!
 

Frank Dernie

Master Contributor
Forum Donor
Joined
Mar 24, 2016
Messages
6,445
Likes
15,778
Location
Oxfordshire
If you did do headphones or speaker testing - maybe it would be a good idea to keep to basic technologies to not get overwhelmed by too much stuff and sort of figure where to go from there? If headphones - perhaps a Ribbon like Raal, a AMT like Hedd, a Graphene like ORA, and a representative Electrostatic and dynamic. I suppose a cheap/reasonably priced pair and expensive pair for each tech - for example to see how a L300 compares to a 009s: does it really sound 10x better since the price is 10x more? (Nice to see how a cheap ESL amp compares to an expensive one while I'm thinking about it!)
If speakers: A good set of regular speakers that people would consider sort of standards? Like different tweeter tech - Say a regular dome/horn or coaxial. Then perhaps something like a Raal ribbon speaker, a Hedd/Adam AMT tweeter speaker, a Magnepan speaker, and a full range non-hybrid ESL like a Quad or Final Audio ESLs/etc. (I've personally got Final Audio ESL's and couldn't go back to anything else) I suppose if you could get your hands on some plasma tweeters - I doubt many people would ever own them but out of curiosity it would be cool to see too!
For me concentrating of a particular tech would be a big mistake. Whilst it can give a marketing department a unique selling point for their product my experience would be that it is the implementation whhich makes the difference, not the technology used.
There are superb sounding conventional 'phones and awful ones. The same can be said of every other technology. What counts is good engineering, and usually a well engineered product using established technology will out perform some boutique technology employed simply to be different and at a much lower cost.
 
Top Bottom