An informal listening test proposition based on Sean Olive’s 2024 NYC CanJam presentation

[MayaTlab]

I have occasionally been using the website FunkyABX as a way to upload and compare a number of EQed files for my own use, and as I had prepared a number of files with some of the newer 5128 / type 4.3 ear simulator targets Harman recently tested (discussion in this thread), I thought that perhaps it could be interesting to share them.

I'd like to mention first and foremost that I don't consider this a properly controlled and designed test, for reasons developed below, but it could still prove useful to some of you to some capacity.



To perform these listening tests, you will need a pair of Truthear Zero Red, as the files are EQed assuming that these will be used as the reproduction device.

In addition :

• If they fit you well, please try to use the narrow bore tips they were provided with.
• No EQ.
• No impedance adapter.
• Plug them into a source with a known output impedance of less than 1ohm.
• Do not take this test by EQing another pair of headphones to the Truthear Red Zero.

Below is a set of links to each test. The targets have been anonymized and given a letter designation (A, B, C, D, E and F) valid across all tests. The tracks noted as “recommended” were used in Harman’s listening tests (but perhaps not the same extract or master).

This WE I’ll provide another link to another test where the targets will be revealed by associating each letter with a label. My advice would be to have performed the test for at least some or all the tracks below by then - but don't feel pressured into taking them all !

Jennifer Warnes - Bird on a Wire (recommended)

Tracy Chapman - Fast Car - Chorus (recommended)

Tracy Chapman - Fast Car - Verse (recommended)

Steely Dan - Cousin Dupree (recommended)

Toto - I Will Remember (recommended)

Alice in Chains - No Excuses - MTV Unplugged (optional)

Daft Punk - Get Lucky - Verse (optional)

Daft Punk - Get Lucky - Chorus (optional)

The Church - Aura (optional)

Grace Jones - Slave to the Rhythm - Ladies and Gentlemen (optional)

Beyoncé - America has a Problem - Single (optional)

Mark Lanegan - Riding the Nightingale (optional)

Yves Tumor - Lovely Sewer (optional)

Slowdive - Star Roving (optional)



Some notes on the EQ process methodology :

• The frequency response of the Truthear Zero Red was obtained by averaging data from several samples. As the 5128 fixture is quite susceptible to produce rocking modes at low frequencies, that may quite likely not be present in people’s ears in most cases, the average that was obtained has been first modified around 100-500Hz to compare it more easily with smoother targets in that region. These modifications were done as seen in the graph here, based on my own experience controlling for it in other test fixtures and other data points. This is to some extent an arbitrary process. I wish some more work had been done already to try to solve this problem, but in the meantime this approach will have to make do.

• Equalisation was rather strictly applied up to around 8kHz, with some attempts at maintaining a close fit to the target up to around 10kHz. Above that a shelf was used to bring the level visually in line with the average. This is not what most would recommend how you should EQ your own sample to a particular target for your own use, but this is closer to the method Harman typically uses.
• A theoretical EQ profile was designed in Room EQ Wizard and then converted to be applied using AUNBandEQ (IIR filters) with Audacity. As there may be some slight variations between how different EQ plugins actually apply the filters, particularly at higher frequencies, an extra filter was added in AUNBandEQ to compensate for any discrepancy that was found (for two targets). The spectrum of the tracks was analysed after applying EQ to make sure that the profile was properly applied. The files were then normalised for loudness according to EBU R128.

• If applicable, the targets were directly digitised after Sean Olive’s latest presentation at Canjam NYC 2024. There were some minor discrepancies between different sources, but I thought that this could be closer to what was tested. A somewhat arbitrary choice had to be made in regard to one target in particular, and it could be interesting to discuss it later on - there are good reasons why I chose it instead of the alternative .

Some notes on the choice of the Truthear Zero Red :

This isn't an ideal choice, far from it. But I can't think of much better alternatives across the board either. If I were to list the positive and negative aspects of that selection, this is what I'd suggest (feel free to complete that list) :

Negatives :

• A poor fit for quite a few people. This can affect the frequency response in situ.
• The frequency response varies with output impedance.
• In an ideally sealed scenario, the front vented design means that the SPL below around 100-200Hz will vary a little bit more in relation to the 200-3k Hz range depending on one’s ear canal volume / impedance, compared to an IEM with a sealed front volume. My sentiment is that this will be swamped by other forms of HPTF variations at higher frequencies, but worth noting nonetheless.

Pluses :

• A large number of samples have been measured by the same operators in the same fixtures, this gives us an idea of its sample variation, which while not perfect seems reasonable.
• The treble response, when measured in a 5128 / type 4.3, is rather smooth above around 8kHz+ and doesn’t show sharp, narrow Q peaks that could be the sign of resonances difficult to successfully EQ for everyone, which may have affected preferences. Given the important inter-individual variability that can be expected in that band, I can't pretend that no peak will occur for every single individual, but as the type 4.3 simulator claims to be a better average of the human ear canal than type 3 fixtures (with a good deal of evidence to support that claim), my sentiment is that choosing an IEM that behaves nicely, in that band, in that fixture, is a better starting point.
• It’s quite tolerant to leakage compared to some other passive IEMs I've compared it to.
• It’s quite cheap and seems to have sold well.

Besides I just own it and originally made the files for my own use anyway .

I’m open to try this with another IEM if you can suggest one that features all of the pluses above and solves the minuses (in particular the fit issues). But most of the usual suspects you can think of, I've probably thought about them too, and have some issues with as well !



Some notes on the listening test format and some suggestions on the questions and conclusions you can and can't draw from it :

I think that it’s important to emphasise that this is not a properly controlled and designed listening test to provide overall data worth analysing. As a consequence I’ve disabled the capability to see the average results as I don’t think that it would lead to particularly fruitful conversations. Think of it rather as a way to provide potentially useful information for you in regards to your preferences (if you own a pair of these IEMs, can it help in steering you towards an EQ profile that will be preferred ?), or more generally a better understanding of what it means to have to express preferences among such close targets when having to deal with other variables (such as program material).

The test format does not adhere to good MUSHRA practices (but should it exactly ? I have some questions on the proper use of anchors for a test such as this one) and I don't think that the rating scale is quite appropriate. There's an API called webMUSHRA that I think could be more appropriate to use but I have no idea how to do so.

I would advise against considering it a blind test. It certainly isn’t for me as I applied the EQ profiles in the first place and can recognise them anyway most of the time, but you must certainly come in knowing how some of these targets, some of which you’ve probably already EQed some of your IEMs to, tend to sound relative to each others.

And I would also be careful about over-interpreting the results you’d get. Among other factors, whether it’s because of sample variation, leakage, varying ear canal length and volume or varying eardrum impedance, the in situ frequency response is uncontrolled. Besides, IEMs may not transfer in a similar way between an ear simulator and your own ears, so what you can take away by EQing your own sample of the Truthear Zero Red to one of these targets may not perfectly translate to other IEMs.

The questions that I think could be more interesting are :

• What is your take on that website’s test format ? Is it helpful for you ?
• Did you choose different targets for the different tracks - in particular the two tests where two different extracts from the same song are selected ?
• Is there any target that struck you as ideal, or did you struggle to express your preference among your favourites ? Do you feel that a mixture of two of these targets, or a more specific EQ adjustment, could lead to a stronger preference for you ?
• If you equalise another IEM to the targets you preferred in this test, do you similarly like the results ?

If you have any questions on or suggestions for that listening test proposition, or have noticed any mistake or error on my end, feel free to comment or reach out.

 

[markanini]

I think the webiste provides a acessible interface, and that's a good thing. I did chose the same target 9/13 times and I could identify it most of the time it after 5-7 songs.

 

[muza_1]

The website is easy to use and straight forward.

I chose the same target for all tracks and got most of the first target right, I had to do another pass for some after posting.

Although I chose the first target I prefer it a little less bright I suspect that the as is the first one would get a bit fatiguing after a while, I am a bit treble sensitive.

After a lot of time experimenting with EQ and different IEMs I found out that my preference is the Harman 2019 target with a Hi-shelf filter @ 2.5kHz to 5kHz of -2 to -5db depending on the IEM (I try to use reliable measurements for EQ and do the final touches by ear).

Edit. My results, plus some edits.

 

[MayaTlab]

Thanks for your participation.

I just want to make it clear that I also do not have access to the overall results, or to any individual's results. You can either share your experience or your preferences, or keep them to yourself, it's entirely up to you and you shouldn't feel obliged to do so.

I'll also share the original non-EQed files, the txt data for the targets that were used, and some spectral analysis plots this WE.

 

[markanini]

Thanks for your participation.

I just want to make it clear that I also do not have access to the overall results, or to any individual's results. You can either share your experience or your preferences, or keep them to yourself, it's entirely up to you.

I'll also share the original non-EQed files, the txt data for the targets that were used, and some spectral analysis plots this WE.

Hopefully as many as possible can share their results in the thread. Here are my results:

Spoiler: Results

Slowdive - Star Roving · FunkyABX

funkyabx.gearpix.app

Yves Tumor - Lovely Sewer · FunkyABX

funkyabx.gearpix.app

Mark Lanegan - Riding the Nightingale · FunkyABX

funkyabx.gearpix.app

Beyoncé - America has a Problem (Single) · FunkyABX

funkyabx.gearpix.app

Grace Jones - Slave to the Rhythm (Ladies and Gentlemen) · FunkyABX

funkyabx.gearpix.app

The Church - Aura · FunkyABX

funkyabx.gearpix.app

Daft Punk - Get lucky (Verse) · FunkyABX

funkyabx.gearpix.app

Daft Punk - Get Lucky (Chorus) · FunkyABX

funkyabx.gearpix.app

Alice in Chains - No Excuses (MTV Unplugged) · FunkyABX

funkyabx.gearpix.app

Toto - I Will Remember · FunkyABX

funkyabx.gearpix.app

Steely Dan - Cousin Dupree · FunkyABX

funkyabx.gearpix.app

Tracy Chapman - Fast Car (Chorus) · FunkyABX

funkyabx.gearpix.app

Jennifer Warnes - Bird on a Wire · FunkyABX

funkyabx.gearpix.app

 

[truckers_atlas]

At risk of looking like more of an idiot than I usually am, here are my results from the test:

Spoiler

Target	Bird on a Wire	Fast Car - Chorus	Fast Car - Verse	Cousin Dupree	Toto	Get Lucky - Verse	Get Lucky - Chorus	America Has A Problem	Average
A	3	3	4	4	5	3	4	4	3.75
B	2	2	2	2	2	2	2	2	2
C	3	3	3	3	4	3	3	4	3.25
D	1	1	1	1	1	1	1	1	1
E	5	4	5	3	5	4	5	3	4.25
F	4	5	5	5	3	5	4	5	4.5

 

[MayaTlab]

Thanks for participating. I don't think that anyone should feel like an idiot coming out of this, and I don't think that anyone will anyway .

 

[MayaTlab]

Link to the "TARGET REVEAL" test :

TARGET REVEAL - Jennifer Warnes - Bird on a Wire



Some notes on how target D was chosen, why the presentation was a bit confusing in that regard, and some potential explanations why it sounds different from target B :

I’ve put this one under the spoiler banner, but this is quite a substantial paragraph and will be useful if you’ve been puzzled as to why targets B and D sound different, even though you may think they shouldn’t.

Spoiler: Notes on target D, and why it sounds different from target B

During the presentation, Sean Olive showed on slide 50 the transfer function between the 5128 fixture and the 711 fixture for 20 IEMs (plotting the 5128 measurements over the 711 ones), and the average :

This average compares quite similarly for most the the spectrum with the average transfer function found by others or using their data for individual IEMs, with some notable differences at higher frequencies and minor differences below which could be explained by, among other factors, different sets of IEMs, or different operators’ practices (for example in terms of insertion depth) :

To transpose the Harman IE 2019 target for 711 fixtures to the 5128 / type 4.3 fixture, one can propose to use the average transfer function, and simply apply it to the Harman IE 2019 target. This is what is shown on slide 52 :

However, as noted during the presentation, the transfer function of the Sennheiser Momentum iei is a little bit different from the average, with two slightly different results from two different operators (Sean Olive and Todd Welti) :

This means that if the average transfer function was used to derive the IE 2019 target for 5128 / type 4.3, the error curve between the Momentum iei and the derived target would be different from the error curve obtained by measuring this IEM in a 711 fixture and comparing it to the IE 2019 target.

It is therefore proposed to use the Momentum iei transfer function as a way to derive the IE 2019 target for 5128 fixtures, for the purpose of a listening test using these IEMs as the reproduction device.

And here’s where it gets confusing to me : the curve shown on slide 54, which is supposed to represent the IE 2019 target derived for 5128 using the Momentum iei transfer function, is inconsistent with that claim. If I apply Sean and Todd’s transfer functions for the Momentum iei to the Harman IE 2019 target for 711 couplers, the curves I obtain are different from the curve obtained by digitising the trace on that slide :

Did I miss anything ? It’s as if the curve obtained from Sean’s transfer had been manually adjusted to realign it with the curve obtained from the transfer function obtained from the average of the 20 IEMs tested above - with actually a tiny bit more energy than the latter in the 200-600 Hz region and less above 2-3k Hz.

We don’t know exactly what EQ profile was applied to the Momentum iei during this test. But if it was measured against the curve obtained from slide 54, then the error curve would be different, which means that after equalisation to this target, it would have sounded different from how the Harman IE 2019 for 711 sounds like when using this IEM as the replicator IEM.

I want to make it quite clear that, regardless of how exactly the Momentum iei was equalised, in no way do I find that particularly important in terms of the conclusions we can draw from Harman’s listening test. Given the likely statistical (in)significance between the top curves’ scores I don’t think that we should conclude anything about a “most preferred target” anyway, but rather conclude that beyond a certain point, alternative targets are close enough, that the results we get are noisy enough, that we should express preferences in terms of probabilities, boundaries or classes, and / or start to better control FR at the eardrum at higher frequencies.

But it means that I had to make a choice between either the Harman IE 2019 for 5128 target shown on slide 52, and the one shown on slide 54. I decided on the latter for three reasons :

a) the Momentum iei’s transfer function between 200-3k Hz is similar to the average and to the Truthear Zero Red’s. If I wanted to get closer to what was actually experienced during Harman’s listening test in that range, then the target on slide 54 would be the one to follow.

b) Around 4-8kHz, the transfer function between 5128 and 711 for the Truthear Zero Red also diverges from the averages obtained, albeit in quite a different - if not opposite - fashion from the Momentum iei :

If I had used the target shown on slide 52, I would have had to add even more energy at higher frequencies, while if the Momentum iei was indeed equalised to the target shown on slide 54, it received less energy in that band than it should have.

c) I suspected that, given the significant boost required at around 4-7kHz to reach the target, it wouldn't perform superbly well, so a more conservative target would have more chances.

So it felt to me that, while still not identical in the slightest, using the target shown on slide 54 would result, for the Truthear Zero Red, in an experience that was closer to how the Momentum iei sounded like to participants during that test, if it was indeed equalised to that target.

If one wants to listen to how these files would have sounded with the target shown on slide 52, feel free to ask.

That being said, generally speaking, if you want to follow that approach and EQ another IEM to Harman IE 2019 for 5128 / type 4.3, I’d recommend using a target that applies the average transfer function from a cohort of IEMs (whether it’s the one obtained by Harman or others, or an average of all of these averages) to the IE 2019 target for 711 couplers, so one similar to the one on slide 52.

So there you go : the reason why targets B and D sound different is the product of the IE 2019 target for 5128 from slide 54 being inexplicably different from what you’d expect, and inconsistent transfer functions across different IEMs between these ear simulators.

I would also like to point out that these transfer function issues also happen between your own ears and the measurements that you read, whether we’re talking about type 4.3 / 5128 fixtures or 711 ones (but likely more so the latter).

Spectral analysis of the test tracks :

The spectrum of a typical musical track is quite steep, therefore to make it easier to compare spectrums, I think that it’s better to try to compensate them one way or another.

I have read a couple of articles on the average spectrum obtained from a large sample of musical tracks, but the only tool I have to analyse a track’s spectrum is Audacity, and I am not confident that the algorithm used to derive the spectrum is exactly the same.

So I’ve arbitrarily chosen here to average the spectrum of around 3min length extracts (representative of the full length’s spectrum - this is a limit from Audacity) from the tracks in that test that Harman also uses, compensate each to that average, and use the four spectrum traces obtained as a reference. This keeps the relative difference between the traces intact, shows that even within the sort of reference tracks Harman uses there are some notable differences, and helps compare each extract to the reference tracks’ spectrum in a way that's easier to read.

Please keep in mind that it is unclear whether or not the master that I used is similar to the one Harman used for their reference tracks.

Here’s a dump for each test track vs the reference tracks and their average :




The tracks Harman uses all follow a similar pattern (broad spectrum), but they still vary a bit. I find it interesting, for example, how different Fast Car is between the verse section and the chorus section. Some of the tracks I added on my own tend to follow this pattern (Alice in Chains for example), others rather intentionally deviate from it (Slowdive).

Looking at some individuals’ results, I am not certain that, beyond a certain degree of convergence, a good correlation between the shape of the spectrum and preference for different targets can be found, but I’d be curious to have your opinion on that matter. I’ve noticed though that at least one track may have resulted in different results for some of you, Mark Lanegan’s Riding the Nightingale. Its spectrum deviates indeed from the reference tracks in a way that could make some of the targets too “hot” in some areas (some emphasis in the presence region for vocals), but some other tracks do as well and may not have resulted in such different preferences.

I’ve attached the original, non-EQ extracts to this post (as mp3 files, if one wants the original WAV files ask me). It could be interesting to have your opinion on what you think of the mix / master for some of them and which ones of them did you find more or less appropriate as listening test tracks.

 

[markanini]

So my preferred target was Soundguys. Not surprising given that I like to listen loud for enjoyment and it has less energy in sensitive frequency ranges. I comforts me somewhat how that target was tied in the Canjam presentation. But I also noticed while taking the tests how I changed my choices the longer time I took for each test, some other targets with more 2-6kHz sounded better initially, but not for a duration longer than a minute. Unless, I suppose, you prefer to listen at a lower volume than me. It it were possible somehow I'd be curious about taking a volume controlled test that maps how preference changes at different listening levels, if at all.

 

[truckers_atlas]

I still can't quite put my finger as to why I rated Harman Beta 2024 as generally a step behind the top-tier targets (Headphones.com, LMG.)

I consistently wished for a treble tone control on the SoundGuys target to turn down the treble. Cymbals, hi-hats, and handclaps felt too prominent and sharp in the mixes.

I think the issue is 8 kHz and up for me with the SoundGuys curve, which seems to be elevated relative to the rest of the frequency range. To wit, I've lost interest in nearly all wired IEMs after getting AirPods Pro 2, which I consider one of the best-tuned IEMs on the market full-stop, and APP2 generally follow the SoundGuys curve up to 8 kHz. After 8 kHz, APP2 take a dip relative to the SoundGuys curve.

While I didn't like the 711 version of Harman IE 2019, it didn't sound that far off from sounding good compared to the above targets. My biggest issue wasn't the elevated ear gain region that tends to be the biggest complaint about Harman IE, but rather the hollow midrange. IE 2019 transposed to the 5128 felt unsalvageable.

I generally enjoyed the selection of music for the listening test. I'd probably swap out America Has A Problem for Cuff It from the Renaissance era since there's a bit too much bass in the former song. Thanks for putting forward this test!