Of course the formula has not really been designed. There is a hypothesis that the factors chosen are those that are the most important, choosing of basic simple, single number, figures of merit for each, and finally an empirically derived statistical fit to how those simple values are convolved into a final figure of merit. It isn't exactly a surprise that the formula is not especially robust.
First off, I haven't had the chance to analyze the data but this reply smacked me upside my head.
This, IMHO, is a significant overreaction to one guy's subjective impressions. You're talking about changing your mindset from believing a science that has reliably predicted preference (to more than a flip-of-a-coin degree) to no longer believing it because one person's subjective opinion didn't align a few times? I always listen first, take notes and then look at the data. In nearly every case I have found points in the data that explained why I heard what I heard. And in cases where I haven't I either don't know what to look for or I heard something that wasn't there. I feel strongly about that.
** Flame suit on... ** I hate to be that guy but have you considered that Amir simply isn't the trained listener it's assumed he is? No disrespect, and lord knows I don't need a scolding and paragraphs on Amir's listening sessions with Harman listed as reference, but let's be real here: you are throwing out the baby with the bathwater by taking Amir's personal subjective reaction in a sighted test over some people's near-lifetime of work to quantify preference based on measurements. The only people I would trust explicitly when it comes to subjective feedback are the people who made the music; decided what it was going to sound like going out the door and hopefully had a very neutral set of speakers they mixed them on. I appreciate reading others' takes as more of a recreation but certainly none that I would put significant stock in; especially if they can't help me correlate it to data. **... Flame suit off **
I am not saying it can all be bottled up in to a single set of charts *for every speaker* because there may very well be cases that kind of step outside the norm (the Philharmonic BMR is an example with its wide horizontal dispersion impacting the predicted in-room response). But I just think you're taking a leap that is too far without trying to look in to the reasons. There may be other data points that explain the differences. And maybe this data isn't as good as it may seem at first glance. Or there simply may be the human factor, which no one (myself included) is impervious of.
What does training have to do with it? The research says preference is the same between trained and untrained.
Again, if you want to go by research, my preference should be the same as majority of people. This has been shown in two rounds of blind testing at Harman (different occasions) where I voted the same as majority in the research.
Same thing as you would be doing when you buy a speaker. It is not like you are going to buy a speaker shuffler and compare the new speaker to a bunch of other ones blind. Would you like to be the first one to evaluate one of these speakers or have me to do that first?
Well, you don't want to go there because I can tell you from experience that trained listeners care far less for superficial aspects of audio products than everyday audiophiles. And I firmly fall in the "don't care" category compared to someone who has bought a new toy, or lacks the experience in any kind of audio evaluation.Was the research based on sighted listening?
It is. It came as a challenge to Consumer Reports which had decided Power Response was the measure of merit. That made Harman speakers not look so good so Sean decided to see if he could develop a better metric and the score we have is that.
That's not the case whatsoever. I have never said that. What I have said is that the factors that bias me are not nearly as strong as what biases you all. So you can't be dismissive of my results out of hand that way.
As I explained to richard, there is nothing wrong with the "data." The disagreement is with a scoring system which until I came around, no one had tested. Blind faith was put on it based on how good other Harman research has been, myself included. My experience even early on though pointed to issues in it and hence the reason I did not adopt it. Prior to this, I was a believer and thought the score would be so good that I would not even need to listen to a speaker. I can't say that anymore.
I'll make a distinction between data mining and what Olive did. Modern data mining is more about looking for correlations in raw data and eliciting the existence of correlations. Olive went into the study with a clear idea about what the correlated data were, and what figures of merit to use. The statistical work was finding the factor to apply to these numbers to yield ordering of preference. This is much more conventional statistical analysis than data mining. "Data mining" seems to have become a catch-all phrase, and I would not like to see it applied when there is arguably much more solid backing to the work than just letting a bunch of data mining tools loose over the data to see what turns up. There is much more science here than that.
Indeed. Here is an example analysis. I went to speaker review index and asked it to show all the speakers with rating of 5.0 to max (6.8). It produced 22 entries (really 23 but one doesn't have a recommendation from me):
