I see where everyone is coming from.
I have no formal background in audio, and I only started measuring speakers a year ago because I wanted to be a more informed reviewer. I have the luxury of being able to readily test many different speakers at my job, so that meant I was able to learn through a lot of trial and error and referencing to existing measurements before I was confident enough to publish my results. Even then, there were still some things I got wrong, and I will probably get other things wrong in the future.
Of course, I'm just a guy with a plastic turntable and a microphone. I use a setup with a total cost of about $200, not a $70,000 automated state-of-the-art machine. I'm not expecting the same kind of accuracy no matter how hard I try. I see why people expect perfection from the klippel. We just don't know yet if we're getting that until we can compare to speakers extensive anechoic published results. The 305P was close to that, but even for that all we had was a smoothed, low res measurement of its predecessor.
I think
@DDF makes a good point that if a speaker seems to measure and perform well below what is expected - as I think is the case with the kali below 500Hz or so - it's fair to reach out to the manufacturer and figure out what's going on before critiquing their design methodology. I don't think that jsut because Kali doesn't have the resources Harman once had doesn't mean they don't know how to design a good speaker. Otherwise we wouldn't have good speakers from any small manufacturer.
Ultimately, speaker designers spend much more time testing, measuring, and critiquing their own speakers than we do as reviewers and readers. I don't know whether that should necessarily stop you from publishing measured data when a speaker measures poorly right away, but it's probably fair to reach out after publishing and not come to an absolute conclusion about the process if the data is well off what's expected. I.e. In my job as a reporter, if something seems amiss with a product we've published an article about, we'll say something like "we're reaching out for more information/clarification and will update this post if we hear back." If they don't get back to us, that's their fault.
A speaker I measured a while ago seemed to have poor measured treble performance and sounded a little muddy. Upon further inspection, it seemed the tweeter had been damaged at some point in shipping - ever so slightly enough to not be noticeable if you're not paying attention. Luckily, I had the other speaker in the pair to compare it to, and the damage was evident in the measurements. I got a replacement from the manufacturer, and it both sounded and measured noticeably better. If I hadn't checked, the speaker would've gotten a bad review.
On the other hand, I'll say it again: The onus in on the manufacturer to provide their measured data and dispell errors in consumer measurements. Whether that's Kali, JBL, or whomever. They have the data, but they don't want to share or can't share for bureaucratic reasons. I don't know any major speaker manufacturer that doesn't measure its speakers, even if they're tuning by ear.
This doesn't just go for bad speakers, it happens with good speakers too. It's something we need to change about audio marketing and culture. Look at my spin of the devialet phantom reactor for instance.
I was pretty shocked. This is a tiny speaker with a pair of 4-inch woofers aimed at the luxury market. But other than an
excess in bass (which it maintains to pretty high SPL levels), it measures better than many studio monitors, let alone hi-fi speakers. It would probably look even better in a proper measurement rig.
Devialet has
no reason to hide this data. They could've built up so much more clout with audiophiles if they shared it. Instead, they're often just perceived as another luxury audio brand that doesn't care that much about sound quality (software issues aside). You can say the same about Sonos, whose few measurements I've seen have been pretty solid.
And yet, hide it, they do. Probably because that's what marketing told them to.
The good thing is Amir's work is helping to change culture around measurements and data provided. There are some growing pains, but I believe it will have the same effect with speakers.
Going forward, I think the solution is simple. No need to continue doubting the uber-expensive measurement rig with years of research behind it. Let's test more speakers with available data. Someone send this man a KH80 or Revel. Then we'll know if it's a speaker issue or a configuration issue. In the meantime, it looks like he'll be getting a replacement for the IN-8, and we'll eventually be getting data from Kali we can compare to. That's ultimately a win for us too.