ZMF Caldera Headphone Review

[isostasy]

I wasn't just eyeballing the graph; I was actually counting the decibel difference. That graph simply doesn't show the bass deficiency (it has more than the Stealth according to that graph) and there's no massive dip at ~4k. The only place that graph and Amir's agree is the ~4db dip around 1.5k. I don't think it's coincidence that's the only place I actually hear a dip. As for the bass, I don't think I agree with that graph that the Caldera is bassier than the Stealth/Aeon, but neither do I think the Stealth/Aeon is ~3-5db bassier than the Caldera as Amir's measurements suggest. They sound fairly close from what I can remember (and from what I can directly compare with the Aeon, which is tuned very close to the Stealth). It's entirely possible that the different pads and seals are dramatically affecting the measurements of both.

You say above "I'm a big believer in science and understand very well the pitfalls of various human cognitive biases".

It's hard enough doing an A/B comparison let alone going by what you remember some music sounding like through a headphone.

You can't expect Amir's graph to "agree" as the measurement rigs are completely different.

"The deviation is very high in the sub-bass (due to non-reliable seal), around 3 kHz (due to the wrong acoustic impedance of the pinna), and in the frequency regions 5 kHz and upwards, where deviations are much to high to obtain reliable measurements from the EARS."

https://www.reddit.com/r/headphones/comments/7szpqm/_/dt9pm7d

 

[amirm]

Has Harman's preference curve data been reproduced or validated by a third party study? I tried to find an answer to this on my own but could not find any information.

Not directly. But a study was published this year on a different fixture, the BK 5128. They used some "harman-like" responses from our site and compared it to one SoundGuys review site uses and some others:
An Over-Ear Headphone Target Curve for Brüel & Kjær Head
And Torso Simulator Type 5128 measurements

Gabriele Ravizza, AES Member1, Julián Villegas, AES Member2, Christer P. Volk, AES Member1, and Tore
Stegenborg-Andersen, AES Member1
1SenseLab, FORCE Technology, Hørsholm, Denmark
2University of Aizu, Aizu Wakamatsu, Japan

They found the Harman-like target to have the highest preference:

"Amongst the five top-rated
frequency curves there are both versions of the B&K
HATS 5128-approximated Harman 2018 curve. This
curve is expected to be globally preferred, with the
second version being the highest-rated curve in our
experiments."

And, this:

".... The fact that the approximated
Harman curve, APHarman2018v2, was the
highest rated amongst participants shows how stable
listener preference is and how reliable perceptual audio
evaluation has become."

On different audience preferences:

"Our results do not support the necessity of creating different
target curves tailored to disparate listener groups
based on preference. This is unexpected, as both previously
published studies (e.g., Olive et al. [10]), and our
own unpublished studies have had clusters among assessors
with true differences in preference, i.e., not just
differences in scale use, discrimination ability or acceptance
level (critical listeners or not). We anticipate that
larger and more diverse studies, when conducted, may
reveal such a necessity."

Study finds once again that bass and treble boost is necessary for listener preference. From the abstract:
"We found that the highest-rated frequency response curves seem to have a boost (< 5 dB) at low
frequencies (< 125 Hz) and a larger boost (5–10 dB) at frequencies between 2–8 kHz."

ZMF Caldera goes against this.

 

[amirm]

@amirm

Sorry if I missed it, but why don't you put the standard disclaimer/note for headphone measurements on every headphone review?

I thought at this point we were past that and folks were fully educated on what my headphone measurements mean. I hate to have disclaimers and only put that in there early because of food fights.

 

[Madlop26]

Is "not perfect but the best we have" good enough evidence for ASR to pressure headphone manufacturers into using it as their design standard?

If you know about statistics, you know that the Harman curve may not be a perfect representation of the general population preference, but Harman has enough sample that the final curve (if we ever get one) will be quite close to it.
You do not sound like a man of science using over the board statement like "pressure headphone manufacturers" ??? Amir reviews headphones and recommended them according to a current a standard, that is, manufactures can do whatever they want, actually from business standpoint to have a precise curve preference is not business sound as limit their options, they can make way more money with cognitive bias, so do not expect too much help from them.

 

[isostasy]

It's more appropriate than you think. "Treatment as usual" might be based in no evidence whatsoever if there is no literature. For the purposes of a study, the control group could be no treatment, but that doesn't necessarily reflect what is being done in clinical practice. If there is no literature for a given intervention, a clinician may choose to act on their best clinical judgment, which may lead to a wide range of non evidence-based interventions, not unlike a wide range of headphone frequency responses.

The heart of my argument is whether "moderate at best" is justification for a public pressure campaign to adapt an industry standard that has been developed by one set of corporate-funded research and has not been validated by a third party. To me, the answer is no.



Yes, I'm aware medical research is time, labor, and money intensive, but I suggested it to illustrate a point: this same charge is leveled at anyone who would question the Harman research on ASR, or even suggest that further research is needed before adopting it as a industry standard. Realistically, Amir may not have the resources, I can't comment on that specifically, but if he wants the Harman curve to be adopted by the industry, an industry that includes manufacturers who are skeptical of its validity, then perhaps he can be the third party to validate the data. Corporations fund research for one reason, and it isn't intellectual curiosity, even more reason the data should be proven to be replicable. For all we know, another organization could release a new preference curve with compelling data in the next five years with a product to follow.

Is "not perfect but the best we have" good enough evidence for ASR to pressure headphone manufacturers into using it as their design standard?

I dispute the existence of "a public pressure campaign". Not to diminish his work but all Amir essentially does in these reviews is plot some graphs, address the errors with EQ, and provide before/after listening impressions, in what I imagine is a very small corner of the internet. What you're saying in this case is you'd continue putting more faith in the previous non-evidence based $3.5k interventions despite the new, as you put it "moderate at best" intervention, being entirely free to try in its cheapest form?

What do you consider currently better than this corporate funded research? The headphone reviewed here has been developed by a single person with no available research backing, corporate or not, and costs $3.5k. If spending $3.5k was the only way to try the Harman target I would sympathize more with your stance, but it's clearly the other way round.

Even then, Amir gave this headphone a golfing panther with EQ for crying out loud and said distortion is so low that this way it is an "excellent sounding headphone that looks great and is comfortable to wear." And "whether it is worth $3,500, you have to decide." would be a valid conclusion for almost anything bar a house or car.

 

[solrage]

You say above "I'm a big believer in science and understand very well the pitfalls of various human cognitive biases".

It's hard enough doing an A/B comparison let alone going by what you remember some music sounding like through a headphone.

You can't expect Amir's graph to "agree" as the measurement rigs are completely different.

"The deviation is very high in the sub-bass (due to non-reliable seal), around 3 kHz (due to the wrong acoustic impedance of the pinna), and in the frequency regions 5 kHz and upwards, where deviations are much to high to obtain reliable measurements from the EARS."

https://www.reddit.com/r/headphones/comments/7szpqm/_/dt9pm7d

And I stand by what I said, but I also have enough experience to know that there's a bit more to audio than just assuming that everything that's measured is heard or everything that's heard is measured. It's a bit more subtle than that. Measurements systems (none of them) hear the way the human ear or brain does, and we have plenty examples thanks to audio science. An example might be how mics pick up all reflections and count that towards volume while our ears filter out reflections to a large degree It's not unreasonable to think there are other kinds of "filtering" happening with headphone listening in which measurement systems pick up on things the ear doesn't and perhaps vice versa.

I absolutely agree about the difficulty of doing a/b comparisons, and I don't expect Amir's graph to agree with other rigs that are different; but my point was that here are two measurements that say completely different things in terms of the relative (forget actual) bass performance of these two headphones. How am I supposed to tell which is more accurate except with my ears? The post you quoted is interesting but it suggests what I already said that difference in seals will affect the bass, but there are still several possibilities that can explain the difference in terms of which measurement got the best/correct seals on each headphone. What you posted also doesn't explain the dramatically difference around the 3-4k range, again speaking relatively. According to Amir's measurement the Caldera should have a huge dip there compared to the Stealth, yet on the other measurement they are nearly equal in that frequency range.

In terms of doing a/b test, I get one can't reliably do it (much less blindly) with actual phones, but I can do it blindly by implementing Amir's EQ and then switching (without looking) between them. I might try that out with the Caldera when I have some time. I already use a low shelf filter and a 1.5k filter and turn them on/off depending on the music (I really feel like I'm EQ'ing the recording as much as the phones). It'll be easy to add the other ones and then switch between them blindly.

 

[isostasy]

And I stand by what I said, but I also have enough experience to know that there's a bit more to audio than just assuming that everything that's measured is heard or everything that's heard is measured. It's a bit more subtle than that. Measurements systems (none of them) hear the way the human ear or brain does, and we have plenty examples thanks to audio science. An example might be how mics pick up all reflections and count that towards volume while our ears filter out reflections to a large degree It's not unreasonable to think there are other kinds of "filtering" happening with headphone listening in which measurement systems pick up on things the ear doesn't and perhaps vice versa.

I absolutely agree about the difficulty of doing a/b comparisons, and I don't expect Amir's graph to agree with other rigs that are different; but my point was that here are two measurements that say completely different things in terms of the relative (forget actual) bass performance of these two headphones. How am I supposed to tell which is more accurate except with my ears? The post you quoted is interesting but it suggests what I already said that difference in seals will affect the bass, but there are still several possibilities that can explain the difference in terms of which measurement got the best/correct seals on each headphone. What you posted also doesn't explain the dramatically difference around the 3-4k range, again speaking relatively. According to Amir's measurement the Caldera should have a huge dip there compared to the Stealth, yet on the other measurement they are nearly equal in that frequency range.

In terms of doing a/b test, I get one can't reliably do it (much less blindly) with actual phones, but I can do it blindly by implementing Amir's EQ and then switching (without looking) between them. I might try that out with the Caldera when I have some time. I already use a low shelf filter and a 1.5k filter and turn them on/off depending on the music (I really feel like I'm EQ'ing the recording as much as the phones). It'll be easy to add the other ones and then switch between them blindly.

Did you click on the link and read the whole post? It does, in fact, fully explain the difference at 4k and he even shows a the difference between his calculated compensation and the provided compensation (which I'm guessing is used in your graph above). Surprise surprise, big difference at 4k.

"How am I supposed to tell which is more accurate except with my ears?" Go by the more accurate measurement (Amir's) rather than hoping a less accurate measurement will better support your opinion.

I would recommend absolutely trying out the EQ (with the same pads) as you speak about at the bottom and seeing which you prefer.

"I really feel like I'm EQ'ing the recording as much as the phones" This hits the nail on the head and is precisely why lots of us like headphones tuned close to Harman, because different music recorded in different studios/venues (some of them well, some terribly) will sound different and you might want to bump up the treble a bit on one, or the bass down on another, etc. This is how I use my subwoofer with my speakers: I don't have it at the same level for all music, some songs I find mixed with more bass than I prefer already so want to turn it down a little. With headphones, this is easier if you have a headphone which gets you close enough to some semblance of "neutral" to begin with.

 

[L0rdGwyn]

Not directly. But a study was published this year on a different fixture, the BK 5128. They used some "harman-like" responses from our site and compared it to one SoundGuys review site uses and some others:
An Over-Ear Headphone Target Curve for Brüel & Kjær Head
And Torso Simulator Type 5128 measurements

Gabriele Ravizza, AES Member1, Julián Villegas, AES Member2, Christer P. Volk, AES Member1, and Tore
Stegenborg-Andersen, AES Member1
1SenseLab, FORCE Technology, Hørsholm, Denmark
2University of Aizu, Aizu Wakamatsu, Japan

They found the Harman-like target to have the highest preference:

"Amongst the five top-rated
frequency curves there are both versions of the B&K
HATS 5128-approximated Harman 2018 curve. This
curve is expected to be globally preferred, with the
second version being the highest-rated curve in our
experiments."

And, this:

".... The fact that the approximated
Harman curve, APHarman2018v2, was the
highest rated amongst participants shows how stable
listener preference is and how reliable perceptual audio
evaluation has become."

On different audience preferences:

"Our results do not support the necessity of creating different
target curves tailored to disparate listener groups
based on preference. This is unexpected, as both previously
published studies (e.g., Olive et al. [10]), and our
own unpublished studies have had clusters among assessors
with true differences in preference, i.e., not just
differences in scale use, discrimination ability or acceptance
level (critical listeners or not). We anticipate that
larger and more diverse studies, when conducted, may
reveal such a necessity."

Study finds once again that bass and treble boost is necessary for listener preference. From the abstract:
"We found that the highest-rated frequency response curves seem to have a boost (< 5 dB) at low
frequencies (< 125 Hz) and a larger boost (5–10 dB) at frequencies between 2–8 kHz."

ZMF Caldera goes against this.

Well that is a step in the right direction, hopefully there is more to come!

If you know about statistics, you know that the Harman curve may not be a perfect representation of the general population preference, but Harman has enough sample that the final curve (if we ever get one) will be quite close to it.
You do not sound like a man of science using over the board statement like "pressure headphone manufacturers" ??? Amir reviews headphones and recommended them according to a current a standard, that is, manufactures can do whatever they want, actually from business standpoint to have a precise curve preference is not business sound as limit their options, they can make way more money with cognitive bias, so do not expect too much help from them.

That language was too strong, I apologize. But the net result is the same. A negative review for a headphone that does not comply well with Harman presumably results in less sales, while a positive review for a headphone that is Harman compliant increases sales. So what is the behavioral change? More headphones that target Harman. You can already see this occurring with the offerings from DCA, even though some of those releases (i.e., Stealth), have been very divisive outside of ASR, subjectively speaking.

I dispute the existence of "a public pressure campaign". Not to diminish his work but all Amir essentially does in these reviews is plot some graphs, address the errors with EQ, and provide before/after listening impressions, in what I imagine is a very small corner of the internet. What you're saying in this case is you'd continue putting more faith in the previous non-evidence based $3.5k interventions despite the new, as you put it "moderate at best" intervention, being entirely free to try in its cheapest form?

What do you consider currently better than this corporate funded research? The headphone reviewed here has been developed by a single person with no available research backing, corporate or not, and costs $3.5k. If spending $3.5k was the only way to try the Harman target I would sympathize more with your stance, but it's clearly the other way round.

Even then, Amir gave this headphone a golfing panther with EQ for crying out loud and said distortion is so low that this way it is an "excellent sounding headphone that looks great and is comfortable to wear." And "whether it is worth $3,500, you have to decide." would be a valid conclusion for almost anything bar a house or car.

Again, I should not have used those words. What I would say is more information is needed before adopting Harman as an industry standard. The golfing panther with EQ is a consolation prize, it's the equivalent of saying "it was broken, but I fixed it!". I don't think many headphone manufacturers are going to be thrilled with that result.

 

[Chagall]

I thought at this point we were past that and folks were fully educated on what my headphone measurements mean. I hate to have disclaimers and only put that in there early because of food fights.

Bold. Think you give people too much credit and your measurements do carry a lot of weight, so like it or not a simple note goes a long way to reduce misunderstandings.

 

[Mulder]

If you know about statistics, you know that the Harman curve may not be a perfect representation of the general population preference, but Harman has enough sample that the final curve (if we ever get one) will be quite close to it.

How close?
How do you know?

 

[amirm]

How close?
How do you know?

Within the different populations they tested, yes. Here is an example:

The error bars are small but are on the plots. I have circled a couple.

 

[amirm]

I'll state it again so nobody reading this confuses responses to my writings as points I actually have been trying to state. It continually is assumed I am anti Harman when that couldn't be further from the truth.

When you are producing a headphone with so much deviation from Harman researched target, it is clear you are against it. How you feel about it outside of that is neither here, nor there. We are buying the headphone, not your thoughts.

 

[amirm]

So, with that being said, if ASR wants to advance the field and create industry guidelines, perhaps Amir should do some science and show us the research is in fact reproducible. ASR has a large enough readership that a Harman target study could be conducted.

??? We are not here to do other people's homework. If ZMF thinks their target is superior or performant, it is their job to do the testing. They don't need us. They don't need anything fancy at all. You develop some EQ, and recruit a few people to do the first order analysis. You can't honestly be asking us to assess the validity of Caldera response. We have done our work in a) measuring it and b) me EQing it and liking the Harman response better.

My unique value is the combination of test gear, knowledge of the same and overall technology, and means to bring that to this forum. You don't need any of that to conduct a simple preference test with EQ AB. Harman has documented it all if you need help but really, if you are building headphones, you better already know or quickly learn how to do this. Your future depends on it. Ours does not. I have plenty of other things to go and test.

Importantly, across some 100 headphones and IEMs, I have validated Harman response to way beyond my own satisfaction. As such, I have no motivation at all to go and do work. Per above, companies making headphones, and those advocating other responses are people who need to do this kind of study.

It is just remarkable to me that no one wants to do this kind of testing. Last food fight we had was on targets for 5128. We had people drawing blood but would refuse to go and test their hypothesis of a target response for that fixture in a controlled test. They spend $40K+ on a test gear and don't know do a bit of homework to determine how to properly use it??? Makes no sense to me.

 

[L0rdGwyn]

You can't honestly be asking us to assess the validity of Caldera response.

You're right, I'm not, I don't know how you came to that conclusion. My point was that someone outside of Harman should validate the results of their work before it is pushed as an industry standard. You described the beginnings of that line of inquiry in post #422.

 

[MacClintock]

It is a group of papers from one corporation, which has not been validated by a third party organization. The "we know what you like better than you do" is how the information is presented by certain members of this forum, not in the research itself.

AES is peer reviewed, as far as I know, and almost all the papers appeared there, so quite some confirmation.

 

[zach915m]

The aim of science is to delineate areas, fostering a collective understanding and advancing shared knowledge through impartial language and objectivity. It involves consistently employing variables, such as products or headphones, in a standardized manner (as seen in amir's reviews; note that practices vary). It encourages audiences to embark on their individual journeys, recognizing the diverse range of variables at play, as each person has unique Head-Related Transfer Function (HRTF) preferences.

I use the Harman target every time I design a headphone and we do have designs that are closer to it like the Atrium Closed and Bokeh in certain pad and mesh combinations, but not because I as the designer feel that's the only adherence there can be.

If you want a Caldera that conforms closest to harman, use the ultra perf pads, gently EQ to the curve in the area (less has to be done) and then EQ the bass up as the ultra perf will cause a roll-off below 70 HZ. I know about 2 out of hundreds of owners who prefer this, but if that's what you're into, it's the easiest way to get there.

there's 6 sets of different pads for the Caldera, and a titanium mesh for the front that helps tune the headphone to your preferences/HRTF

When you are producing a headphone with so much deviation from Harman researched target, it is clear you are against it. How you feel about it outside of that is neither here, nor there. We are buying the headphone, not your thoughts.

Also with @amirm calling things "errors" vs "deviation" it furthers the ideas that there is a "right" and a "wrong" vs a preference as a guideline

Here's the stock pads vs ultra perf which I mentioned earlier are closer to Harman. If needed, you can EQ the bass up to Harman since it's so low and bring down the 8khz peak. I opted for that instead of using other methods to boost the bass since that would add distortion which I spoke about earlier. I find low distortion plus EQ sounds better to my ear than high distortion and Harman tuning, but I also prefer my stock tuning to an EQ'd one so that needs to be taken into consideration.

Ultra Perf is the dotted line, mantle titan mesh can also be used to bring the treble down if desired:

Also as I mentioned I tuned the Bokeh to Harman in one format, while other tunings with different pads and mesh don't align to Harman.

Here's the tuning that most owners seem to prefer for the BOKEH so far in stock form:

Here's the BOKEH in the closest to Harman format with hybrid suede pads and perf titan mesh:

I can see how with a small sample size of my headphones you may define my perspective as being for or against Harman. But I believe it's not as simple as that and that different users prefer different things at different times, and even sometimes the same users prefer variances at different times, so that's how I've designed our line-up and each headphone. I hope that makes sense.

 

[zach915m]

We have done our work in a) measuring it and b) me EQing it and liking the Harman response better.

Because of this thread I have people emailing me telling me they tried it and like the stock tuning better. They are obviously bias because they own the headphone and are emailing me.

There's also people I'm sure out there that are trying the EQ and liking it better, they also have a bias to this website.

There are also people biased by this review who have not heard the headphones and have made a finite judgement of it.

I also have a bias because I made the headphone and tuned it to my preference for the system.

I also have a bias towards liking the Harman curve the best in the BOKEH config which costs less than 1/3 of this headphone.

You develop some EQ, and recruit a few people to do the first order analysis.

Ofcourse. As ZMF grows and we have the resources for it, we will have time to develop studies to answer why audiophiles like varied responses that can deviate from Harman if someone doesn't do more great research in the mean time. The future is exciting as I've mentioned a few times. Hardcore headphone audiophile research towards more esoteric systems and headphones has not been done yet. The research we have is general, and it's very good.

It is not my policy to ask manufacturers for feedback prior to reviews when members send in products.

Is that right? So you just made a story about being busy about playing with your new HATS instead of telling me this?

Regardless, you don't need to send me anything. As you see, I get gear from members. The advantage in sending me gear is that I give you a preview of any issues I see. If you don't value that, that is cool.

You should be conducting all measurements without talking to manufacturers to give them a "preview." I pointed out how the bias shows up in your reviews and carries over to the members of the site. I noticed this before this review and after we had correspondence and is why I stopped communication with you.

My personal subjective remarks are taken with a grain of salt by the membership.

I hope this is true, but I'm not sure there's unbiased peer reviewed research on it yet. Or why your members need this subjectivity if they are not affected by it.

 

[MacClintock]

I find low distortion plus EQ sounds better to my ear than high distortion and Harman tuning,

That sounds like a total pretext, as the measured distortion in the bass is very low, a nice characteristics of this headphone, so it could without problems have been tuned to have more bass and still low distortion there.

but I also prefer my stock tuning to an EQ'd one so that needs to be taken into consideration.

So again, this does not sound like you prefer the Harman target.

Here's the BOKEH in the closest to Harman format with hybrid suede pads and perf titan mesh:
View attachment 333523
I can see how with a small sample size of my headphones you may define my perspective as being for or against Harman. But I believe it's not as simple as that and that different users prefer different things at different times, and even sometimes the same users prefer variances at different times, so that's how I've designed our line-up and each headphone. I hope that makes sense.

That in fact looks much closer than anything else from ZMF I have seen so far. So why don't you send the BOKEH in in this configuration for measurements?

 

[zach915m]

so it could without problems have been tuned to have more bass and still low distortion there.

In an open system the ways to tune to Harman that I know of add distortion and can be seen in other open planars measured by this site. EQing in this area with something like a Lokius or parametric EQ will have less distortion than tuning to Harman off the bat with those methods and will keep the headphone much easier to drive, and also keep the transient response cleaner. Again if Harman is the goal.

 

[amirm]

I pointed out how the bias shows up in your reviews and carries over to the members of the site. I noticed this before this review and after we had correspondence and is why I stopped communication with you.

I don't believe that story. Nor do I believe the assertion. The proof is right here. I could have bashed your headphone to death, disparaged it for being expensive and non-compliant with a proper target and leave it at that. Instead, I made an EQ for it and then gave it the highest praise possible, more than a number of members here give it.

I am pretty sure the real story is that you noticed that we value the Harman target, realized you were not compliant with it and that you would not get a good review so decided to ghost me. You should know that none of this impacts me other than to point out that you were non responsive to me. I hold no bias and pride myself in acting ethically and professionally. Build good products by our and research standards and you will get high praise.